Tokenization in LLMs

AI / ML Intermediate

debt(d7/e3/b5/t7)

d7 Detectability Operational debt — how invisible misuse is to your safety net

Closest to 'only careful code review or runtime testing' (d7). The detection_hints.tools list is empty; no linter or compiler catches token miscounts. The error typically surfaces only at runtime when a request fails with a context-length error, or silently when content is truncated — often not until testing with realistic payloads. No standard static analysis tool flags word-count-based estimates.

e3 Effort Remediation debt — work required to fix once spotted

Closest to 'simple parameterised fix' (e3). The quick_fix states: count tokens using the provider's API or library (tiktoken / Anthropic count_tokens endpoint) before sending. This is a small, localised change — replacing ad-hoc estimation with a single library call — but it may need to be applied in multiple prompt-construction sites across a feature, making it slightly more than a one-liner.

b5 Burden Structural debt — long-term weight of choosing wrong

Closest to 'persistent productivity tax' (b5). Token budget awareness must be maintained across every LLM call: system prompts, conversation history, and user content all count. Any developer building or extending LLM features must keep this in mind continuously, and the constraint shapes prompt design, chunking strategies, and cost estimates throughout the feature's lifetime. It is not just one isolated component.

t7 Trap Cognitive debt — how counter-intuitive correct behaviour is

Closest to 'serious trap' (t7). The misconception field is explicit: 'Token count roughly equals word count.' This directly contradicts a natural and reasonable intuition. Code and non-English text can be 2–3× more expensive than expected, and the full message array (not just the latest message) is counted — both documented gotchas that contradict how most developers initially model the concept. This is worse than a minor edge case but stops short of 'the obvious way is always wrong' (t9) since English prose is close to the 1:1.3 ratio.

About DEBT scoring → scored by claude-sonnet-4-6 · 2026-05-11 · reviewed by human

Also Known As

token tokens LLM tokens subword tokenization BPE tokenization

TL;DR

The process of splitting text into tokens — subword units that LLMs process — which directly determines context window usage, cost, and model behaviour on non-English and code inputs.

Explanation

LLMs do not process characters or words — they process tokens, which are subword units learned from training data. Common English words are typically single tokens; rare words, non-English text, and code identifiers split into multiple tokens. GPT-4 and Claude use roughly 1 token per 4 characters for typical English text, but code, JSON, and non-Latin scripts are significantly less efficient — a 500-character PHP function might consume 200 tokens while the same function in Thai might use 400. Tokenization affects context window usage (models have hard token limits), API cost (billing is per token), and model behaviour (models see tokens not characters — character-level tasks like counting letters or reversing strings are difficult because the model cannot see individual characters).

Common Misconception

✗ Token count roughly equals word count. Tokens are not words — common English text averages about 1.3 words per token, but code and non-English text can average 0.5 words per token or worse. Always use the provider's tokenizer library to count tokens accurately before assuming your content fits within the context window.

Why It Matters

Misunderstanding tokenization leads to context window overflows, unexpected API costs, and subtle model behaviour issues. A PHP developer building an LLM feature who assumes 1000 words = 1000 tokens may design prompts that exceed the context limit by 40% when code and JSON are included. Accurate token counting before sending requests prevents truncation errors and avoids billing surprises — both tiktoken (OpenAI) and Anthropic's token counting APIs enable this.

Common Mistakes

Estimating tokens from character or word count rather than using the actual tokenizer — always count programmatically.
Not accounting for the system prompt and conversation history in the token budget — the full message array is tokenized, not just the latest user message.
Assuming token limits are symmetric — a 200k token context window does not mean 200k tokens of useful reasoning; models degrade in quality on very long contexts.
Ignoring that non-English content and code consume significantly more tokens per character than English prose.

Code Examples

✗ Vulnerable

// ❌ Truncating by character count — splits mid-token, wastes context
function truncateForLLM(string $text, int $maxChars = 4000): string {
    return substr($text, 0, $maxChars);
    // 4000 chars ≠ 4000 tokens. May cut mid-word.
    // A PHP file with lots of $variables tokenizes very differently than prose.
}

✓ Fixed

// ✅ Count tokens before sending — use the provider's token counter
// Anthropic PHP SDK
use Anthropic\Client;

$client   = Anthropic::client(getenv('ANTHROPIC_API_KEY'));
$response = $client->messages()->countTokens([
    'model'    => 'claude-sonnet-4-20250514',
    'messages' => [['role' => 'user', 'content' => $text]],
]);

$tokenCount = $response->inputTokens;

// Safe truncation: use the provider's tokenizer, not character count
if ($tokenCount > 180000) { // Leave headroom in 200k context window
    // Use semantic chunking or summarise sections
    $text = summariseToFit($text, maxTokens: 150000);
}

// Rule: ~4 chars/token for English prose; code and non-Latin script differ significantly

References

https://platform.openai.com/tokenizer