Tag: tokenization
Search Query Parsing
PHP 8.0+
Transforming raw user search input into structured components - tokens, fields, operators, and phrases - before query execution.
2d ago
search intermediate
Tokenization in LLMs
3
The process of splitting text into tokens — subword units that LLMs process — which directly determines context window usage, cost, and model behaviour on non-English and code inputs.
3mo ago
ai_ml intermediate