Prompt caching: the cheapest speedup you are not using
Providers can cache a stable prompt prefix so repeated tokens are billed at a fraction of the normal rate and processed faster. Capturing that requires putting unchanging content first and variable content last, and keeping the prefix byte-identical across requests.
Most production prompts are mostly the same every time. The instructions do not change. The tool definitions do not change. The document being discussed does not change across a conversation. Only the last part — the user's actual message — differs.
Without caching you pay full price to process that identical prefix on every single request.
How it works
The provider keeps the computed state for a prefix it has seen. On a later request beginning with exactly the same tokens, it reuses that state instead of recomputing it, and bills the cached portion at a large discount.
The saving is real on both axes: less money and noticeably lower time to first token, which users feel more than they feel the bill.
The constraint is exactness
A prefix matches or it does not. One different character anywhere in it and the cache misses entirely — there is no partial credit for being nearly the same.
Order the prompt for it
This is the whole design change. Stable content first: system instructions, tool schemas, style guides, the document under discussion. Variable content last: the current question, the latest turn, anything per-request.
Many prompts are written in the opposite order by accident, with a greeting or a context line containing today's date at the very top. Moving that one line to the bottom can take a hit rate from zero to near-total.
What quietly breaks it
Timestamps. Request IDs. A user's name interpolated into the system prompt. Randomly ordered JSON keys from a map that does not guarantee order. A "conversation started at" line.
All of them look harmless and all of them sit early, which is the worst place for them.
Verify rather than assume
Hash your prefix and log the hash. If it differs between two requests that should have matched, you have found the bug in one step.
Then watch the cache metrics the provider reports. Teams routinely believe caching is working for months while paying full price — it fails silently, because a miss is just a normal request.
Frequently asked questions
Why is my cache hit rate near zero?
Almost always something variable near the start — a timestamp, a session ID, a user name interpolated into the system prompt, or serialisation that does not preserve key order. Log the prefix hash across requests and the culprit appears immediately.
Is caching worth it for short prompts?
Usually not — most providers have a minimum length below which nothing is cached. It pays off when you have a substantial fixed preamble: long instructions, a big tool schema, a document being asked about repeatedly.
