AI API Cost Management
debt(d8/e5/b6/t6)
Closest to 'silent in production until users hit it' (d9), -1 because you can detect cost issues via billing dashboards and token usage logs after deployment, but there are no automated code-level tools (detection_hints.automated: no). Cost overruns manifest as unexpected bills, not code errors. The code_pattern hints (no token logging, no caching, wrong model selection) are not caught by any linter, SAST, or compiler — only manual review or production monitoring reveals them.
Closest to 'touches multiple files / significant refactor in one component' (e5). The quick_fix mentions caching responses, swapping models, and logging token usage — these touch multiple call sites across the codebase. Adding a caching layer with hash keys, routing logic for model selection, and token logging instrumentation across web/cli/queue contexts requires changes to multiple files and potentially introducing new infrastructure (cache store, logging pipeline). Not architectural rework, but more than a simple parameterised fix.
Closest to 'persistent productivity tax' (b5), +1 to b6. AI cost management applies across all contexts (web, cli, queue-worker) and affects every AI feature. Every new AI-powered feature requires model selection decisions, caching strategy, token budget planning, and ongoing monitoring. It's a persistent cross-cutting concern that shapes how every AI integration is built, but it doesn't define the entire system's architecture — it's a tax on all AI-related work streams.
Closest to 'serious trap' (t7), -1 to t6. The misconception is that 'using the most capable model is always best' — developers coming from non-AI backgrounds naturally assume the best model yields the best results, not realizing a 15x cost difference can exist with identical accuracy for simple tasks. This contradicts the general software intuition of 'use the best tool available.' It's a notable-to-serious trap because the 'obvious' approach (use the flagship model) silently works perfectly while hemorrhaging money, but experienced developers learn this relatively quickly once exposed to billing data.
Also Known As
TL;DR
Explanation
LLM API costs scale with tokens (input + output). Optimisation strategies: prompt caching (Anthropic caches repeated system prompts — 90% cheaper for cached tokens), model tiering (use Haiku/GPT-3.5 for classification, Sonnet/GPT-4 for complex reasoning), output length control (be specific about desired length), context minimisation (RAG over full context), batching (async batch API is 50% cheaper), and result caching (cache identical prompt+response pairs in Redis with TTL). Track costs per feature — AI spend can be 10-100x surprising without monitoring.
Common Misconception
Why It Matters
Common Mistakes
- Using claude-opus for every call including simple tasks — Haiku is 15x cheaper for classification.
- Sending full conversation history on every turn — summarise older turns to reduce input tokens.
- No response caching — identical prompts (FAQ answers, product descriptions) should be cached.
- Not setting max_tokens — models default to maximum output length, increasing cost.
Avoid When
- Sending full conversation history on every turn without summarisation — token costs grow linearly with context length.
- Using the largest model for every task — use smaller, cheaper models for classification and routing.
- Skipping caching for identical or near-identical prompts — semantic caching can cut repeated costs dramatically.
- Streaming full responses when only a structured extract is needed — generate only what the application consumes.
When To Use
- Cache identical prompts and their responses — provider-level prompt caching reduces cost on repeated inputs.
- Route tasks to the cheapest model that meets quality requirements — reserve large models for complex reasoning.
- Truncate or summarise long contexts before sending — a shorter prompt is a cheaper prompt.
- Set max_tokens limits on every call — unbounded generation is unbounded cost.
Code Examples
// Every request uses largest model with full history:
$response = $claude->messages->create([
'model' => 'claude-opus-4-5', // Most expensive
'max_tokens' => 4096, // Maximum output
'messages' => $fullHistory, // 50 turns of history
]);
// $0.15 per 1K input tokens * 10K tokens = $1.50 per request// Right-sized model + caching + token control:
// Check cache first:
$cacheKey = 'ai:' . md5($prompt);
if ($cached = $redis->get($cacheKey)) return json_decode($cached);
// Simple task: use small model
$model = $this->isComplexTask($prompt) ? 'claude-sonnet-4-6' : 'claude-haiku-4-5-20251001';
$response = $claude->messages->create([
'model' => $model,
'max_tokens' => 256, // Constrain output length
'system' => $systemPrompt, // Cached by Anthropic after first call
'messages' => $recentHistory, // Last 5 turns only, not all 50
]);
$redis->setex($cacheKey, 3600, json_encode($response->content));