Unicode Normalisation Attack
debt(d7/e3/b5/t7)
Closest to 'only careful code review or runtime testing' (d7). The detection_hints note automated=no, and while phpstan and intl-extension are listed as tools, the code_pattern describes subtle runtime failures (duplicate checks failing for visually identical strings, comparison failures on accented characters) that won't surface until real user data exposes the inconsistency. No standard linter rule catches missing normalisation calls; it requires deliberate review or targeted testing with Unicode edge-case inputs.
Closest to 'simple parameterised fix' (e3). The quick_fix points to adding Normalizer::normalize() calls before comparisons and storage. This is more than a single-line swap because common_mistakes span multiple concerns (username checks, XSS filters, file paths, DB storage), meaning the fix needs to be applied consistently at several input-handling points within the application, but it doesn't require cross-cutting architectural rework — it's a targeted pattern replacement in input-handling code.
Closest to 'persistent productivity tax' (b5). The applies_to covers both web and cli contexts, meaning every place that handles user-supplied strings must account for normalisation. Tags include internationalisation, encoding, and injection, indicating it touches multiple concerns. Every future developer adding a new input handler must remember to normalise, imposing an ongoing tax across many work streams, though it doesn't fully reshape the architecture.
Closest to 'serious trap — contradicts how a similar concept works elsewhere' (t7). The misconception explicitly states developers believe Unicode is handled consistently across PHP string functions and the database. A competent developer accustomed to simple string equality naturally assumes visually identical strings are byte-identical; the reality that NFC vs NFD vs NFKC produce different byte sequences for the same rendered character directly contradicts that mental model. Common mistakes (XSS filters missing Unicode angle-bracket alternatives, filesystem path bypass, uniqueness check failures) confirm the trap is serious and multi-dimensional.
Also Known As
TL;DR
Explanation
Unicode defines multiple normalisation forms (NFC, NFD, NFKC, NFKD). A character like 'é' can be represented as a single codepoint U+00E9 or as 'e' plus a combining accent U+0301. A blocklist filter checking raw bytes may miss the decomposed form, which later normalises to the blocked character after storage or rendering. PHP's intl extension provides Normalizer::normalize() to canonicalise input before validation. Always normalise user input to NFC before validation and storage; never rely solely on byte-level comparisons for security-sensitive checks.
Common Misconception
Why It Matters
Common Mistakes
- Comparing usernames or email addresses for uniqueness without normalising to NFC or NFKC first.
- XSS filters that match ASCII angle brackets but miss Unicode alternatives that browsers render identically.
- File path security checks that pass but the filesystem resolves to a different path via Unicode canonicalization.
- Not using Normalizer::normalize() before storing or comparing user-supplied strings containing Unicode.
Avoid When
- Do not skip normalisation for filenames — different Unicode representations of the same filename can bypass path traversal filters.
- Do not compare usernames or emails for uniqueness without first normalising — homograph attacks exploit visually identical but byte-different characters.
When To Use
- Normalise all user-supplied strings to NFC form before comparison, storage, or security checks.
- Apply unicode normalisation before checking string equality — 'café' composed vs decomposed are different byte sequences but the same character.
Code Examples
// Different Unicode representations of same visual character
// Filter applied before normalisation — bypass possible
$input = $_POST['username'];
if (preg_match('/[<>"\']/u', $input)) abort(400);
// Attacker sends NFD form of < → passes filter → XSS in output// Normalise to NFC BEFORE any validation
$input = Normalizer::normalize($_POST['username'] ?? '', Normalizer::FORM_C);
// Now validate on canonical representation
if (!preg_match('/^[\p{L}\p{N}._@-]+$/u', $input)) abort(400);
// Prevent homograph attacks (visually identical chars from different scripts):
// 'a' (U+0061 Latin) vs 'а' (U+0430 Cyrillic) — restrict to expected script