If you're building an LLM-powered app that sends the same system prompt, tool definitions, or long context on every request, prompt caching is the single highest-leverage optimization you can make.
What gets cached
Most providers cache at the token level, starting from the beginning of the prompt. A typical structure looks like:
[system prompt] [tool definitions] [conversation history] [new user message]
If the first three sections are identical to a previous request, the provider can reuse the internal computation for those tokens instead of reprocessing them — you pay a much lower rate for the cached portion.
Why order matters
Caching is prefix-based. Put the parts of your prompt that change most often (user messages) at the end, and the stable parts (system instructions, tool schemas, long reference documents) at the beginning.
Stable, reused across requests → System prompt, tool defs, docs
Changes every request → User's latest messageIf you interleave stable and dynamic content, you break the cache prefix and lose the benefit entirely.
A concrete example
An agent that re-sends a 2,000-token tool definition block on every turn of a 20-turn conversation is paying full price for those 2,000 tokens twenty times over — 40,000 billed tokens for content that never changed. With caching, only the first turn pays full price; the remaining nineteen pay the cached rate.
When it's not worth it
- Single-shot requests with no repeated context get no benefit.
- Prompts that change substantially between requests (no stable prefix) won't cache well.
- Very short prompts may not clear the minimum token threshold some providers require.
Practical checklist
- Structure prompts with static content first, dynamic content last.
- Keep system prompts and tool definitions byte-for-byte identical across requests when possible.
- Monitor your provider's cache-hit metrics, not just token counts — a low hit rate usually means something in your "stable" section is silently changing.
