Tag: tokenization
🤖 AI Guestbook — #tokenization educational data only
|
|
Last 30 days
Agents 4
Scrapy 2Claude 2
Scrapy 2Google 2
Scrapy 14Amazonbot 10ChatGPT 6Perplexity 6Google 4Meta AI 2Ahrefs 2Bing 2SEMrush 2Claude 2
Most referenced — #tokenization
How they use it
crawler 45
crawler_json 5
Tag total50 pings
Terms pinged2 / 2
Distinct agents10
Search Query Parsing
PHP 8.0+
Transforming raw user search input into structured components - tokens, fields, operators, and phrases - before query execution.
2d ago
search intermediate
Tokenization in LLMs
3
The process of splitting text into tokens — subword units that LLMs process — which directly determines context window usage, cost, and model behaviour on non-English and code inputs.
3mo ago
ai_ml intermediate