Unicode Fundamentals
debt(d5/e3/b7/t7)
Closest to 'specialist tool catches it' (d5). The detection_hints list phpstan and phpcs as the tools, and the code_pattern explicitly calls out strlen(), strtolower(), substr() on multibyte text — these are catchable by specialist static analysis tools but not by the compiler or default linting passes.
Closest to 'simple parameterised fix' (e3). The quick_fix is a pattern-level swap: replace strlen→mb_strlen, strtolower→mb_strtolower, substr→mb_substr. However, the common_mistakes also include MySQL utf8→utf8mb4 migration and adding mb_internal_encoding(), which pushes slightly beyond a single one-line patch into a small consistent refactor, landing at e3.
Closest to 'strong gravitational pull' (b7). The term applies_to all contexts (web, cli, queue-worker) across PHP 5.0+. Every developer touching user-supplied text must keep the mb_* discipline in mind; string handling decisions ripple through form inputs, database columns, API responses, and logging. The MySQL utf8 vs utf8mb4 issue further means the schema itself is shaped by this choice.
Closest to 'serious trap' (t7). The misconception field states developers conflate UTF-8 and Unicode (the encoding vs. the character set). Beyond that, PHP's byte-based string functions silently return wrong results — strlen('ñ') returning 2 looks correct until it doesn't, MySQL's utf8 charset silently dropping emoji, and Unicode normalisation forms causing invisible inequality. These contradict how developers familiar with ASCII-only or other languages expect strings to work, qualifying as a serious, multi-layered trap.
Also Known As
TL;DR
Explanation
Unicode assigns every character a unique code point (U+0041 = 'A', U+1F600 = 😀). UTF-8 encodes code points as 1-4 bytes: ASCII characters use 1 byte (backwards compatible), Latin extended use 2 bytes, most CJK characters use 3 bytes, emoji use 4 bytes. PHP's string functions operate on bytes, not characters — strlen('😀') returns 4, not 1. Always use mb_* functions for Unicode strings. MySQL's utf8 is not true UTF-8 (only 3-byte sequences); use utf8mb4 for emoji support.
Common Misconception
Why It Matters
Common Mistakes
- Using strlen() instead of mb_strlen() for character counts.
- MySQL utf8 charset — only supports 3-byte UTF-8; use utf8mb4 to store emoji and supplementary characters.
- Not setting mb_internal_encoding('UTF-8') — mb_* functions use a default that may not be UTF-8.
- Not normalising Unicode before comparison — 'é' can be one code point (U+00E9) or two (e + combining accent); they look identical but are not equal.
Code Examples
// Byte functions on Unicode — wrong results:
$name = 'José'; // 5 visible chars, 6 UTF-8 bytes ('é' = 2 bytes)
echo strlen($name); // 6 — byte count, not char count
echo substr($name, 0, 4); // 'Jos' + broken 'é' byte — corrupted string
echo strtoupper($name); // 'JOSé' — 'é' not uppercased to 'É'// Multibyte-aware functions:
mb_internal_encoding('UTF-8');
$name = 'José';
echo mb_strlen($name); // 4 — character count
echo mb_substr($name, 0, 4); // 'José' — correct
echo mb_strtoupper($name); // 'JOSÉ' — 'É' correct
// MySQL: utf8mb4 for full Unicode:
CREATE TABLE users (name VARCHAR(100) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci);