Intro
There's a pitch behind every new AI-powered feature: it understands anything a user throws at it. Open-ended, flexible, genuinely intelligent.
The pitch is half true. What it leaves out is that in most production systems, "anything a user throws at it" collapses into the same twenty questions asked twenty different ways. A support bot doesn't get asked infinite variations of physics, it gets asked "what's your return policy," "how do I get a refund," and "can I send this back" on a loop, forever.
That gap, between the imagined variety of user queries and the actual repetition underneath it, is where the token bill lives.
The naive approach: every request hits the model
Say a team ships a support assistant. Every user message goes straight to a frontier model: embed the question, retrieve context, generate an answer, return it. It works. It's also expensive in a very specific way, identical intent, paid for from scratch, every single time.
(Illustrative, not a specific case, but recognizable to anyone who's watched an LLM API bill by day one versus day thirty.)
Exact-match caching almost works, then doesn't
The obvious fix is a cache: hash the incoming query, check for a hit, skip the model call if found. This works exactly once, for the exact same string. "How do I reset my password" and "forgot my password, help" are the same question and two different cache keys. Exact-match caching is a solution for a problem language doesn't actually have, users asking things identically.
Embeddings turn paraphrase into a solvable problem
Semantic caching replaces the hash with a vector. Embed the incoming query, run a nearest-neighbor search against previously answered queries, and if the closest match clears a similarity threshold, serve the cached response instead of calling the model at all. Below the threshold, call the model as normal and add the new query/response pair to the cache for next time.
This is the actual mechanism at work: not "the AI understood the question was similar," but a distance calculation in vector space with a cutoff you chose.
A stale cache is worse than no cache
Here's the failure mode nobody budgets for. A support answer gets cached in March. The return policy changes in April. The cache doesn't know that, it just has a vector close enough to keep matching, and it will confidently serve outdated information for as long as the entry lives. A cache without an expiration policy isn't a cost optimization, it's a bug that pays for itself to keep running.
Threshold tuning is the whole game
Set the similarity threshold too loose and you'll serve March's return policy answer to an April refund question, because the two "look" adjacent in vector space even though the real answer changed. Set it too tight and you get single-digit hit rates, because no two users phrase anything identically. There's no universal number here, only a number you have to measure against your own traffic.
Build it yourself, or don't
If you're at the scale where cache hit rate is a line item on your infra bill, and your domain has stable, well-defined intents (support, FAQ, internal tooling), rolling your own semantic cache with something like pgvector or Redis is a weekend project, not a research problem. If your queries are genuinely open-ended and low-repeat (creative generation, one-off analysis), you're optimizing for a hit rate that doesn't exist, and the caching layer is wasted engineering effort.
Semantic caching isn't really an AI technique. It's ordinary distributed-systems caching (TTLs, invalidation, cache keys) with an embedding standing in for the key. The AI part was never the hard part.
We ran into a version of this question building Cyclopt Companion's analyzers: a one-line diff shouldn't necessarily trigger a full re-analysis from scratch, and figuring out what counts as "close enough to skip" turned out to be the actual engineering problem, not the analysis itself.
What's your current cache hit rate on user-facing LLM calls, and have you actually measured it, or is it a guess?
Top comments (6)
The staleness point is the one that quietly ends most semantic-cache projects. Threshold tuning at least fails loudly-ish — you can measure false-hit rate on a labeled set. Staleness fails on a delay: the March answer keeps matching in April and the vector distance has no idea the ground truth moved underneath it. The fix that actually held for us wasn't a TTL (too blunt — evicts fresh entries, keeps stale ones that happen to be young), it was tying eviction to the source: when the return-policy doc changes, invalidate every cache entry whose retrieval touched that doc. Cache key includes a content hash of the sources, not just the query vector.
The other thing worth caching is refusals — "we don't support that" is high-repetition and cheap to serve, but people forget to cache the negative path. How are you measuring the false-positive rate on hits in practice? A shadow call to the model on a sampled % of cache hits is the only honest way I've found to know your threshold is still right.
Yeah, source-hash in the cache key is the right move, TTLs are just a slow admission that you don't know when your data changed. We ended up somewhere similar: the cache entry stores the set of source IDs + content hashes it was built from, and any write to a source fans out an invalidation. TTLs stay as a backstop for stuff we forgot to wire up, not the primary mechanism. Good call on caching refusals too, "not supported" is genuinely one of the highest-repetition responses, and nobody treats it as a first-class cached path. On measurement: shadow calls on a sampled % of hits, exactly. We log the cached response, the fresh model response, and a cheap similarity score between them; anything below the threshold gets flagged for review. Not free, but the only way I've found to know whether the threshold is still doing its job or just aging into a lie.
There's a subtler cousin of the staleness bug: recency inversion. The moment you fix stale entries by preferring newer ones, you've built the opposite bug - newer is not automatically truer. A rollback, a revert, a "we tried the new policy and went back" all produce a newer entry that is wrong. Timestamps are a tiebreaker of last resort, not a validity signal. What held up for us is explicit lineage: an entry can name the entry it replaces, and when two entries are near-identical and neither names the other, the read path surfaces both with their dates instead of silently picking one.
On measurement, Max's shadow-call sampling is the honest instrument, with one refinement: don't sample uniformly. Oversample the entries whose sources changed since caching - that's where the false hits live, and a uniform sample mostly re-measures the easy majority.
Recency inversion is a great name for it, and yeah, we got bitten by exactly that, rolled a policy back, the "old" answer was suddenly correct again, and the cache confidently kept serving the interim one because it was newer. Lineage links help a lot; we do something similar, where an entry can explicitly supersede another, and ambiguous cases (near-duplicates with no link) get surfaced rather than resolved by timestamp. The oversampling point is the one I hadn't thought about carefully enough. Uniform sampling is basically re-verifying the stable majority every night, while the actually risky entries (sources changed, threshold-borderline hits, high-traffic keys) get the same coverage as everything else. Stratifying the shadow sample by "how much has the world moved under this entry since we cached it" seems obviously right in retrospect. Stealing that.
The stale-cache point is the real trap. I’d also treat knowledge/version context as part of the cache key, not just semantic similarity. Two questions can be nearly identical while the underlying policy or state has changed. Otherwise a high hit rate can quietly become a correctness metric in disguise.
Hard agree. Hit rate flattering itself into a correctness signal is exactly how these systems rot silently. The version-in-the-key framing is cleaner than how I phrased it in the post: it reframes the cache from "same question, same answer" to "same question against the same world, same answer," which is the invariant you actually want. In practice we bind entries to a version vector of the sources they depended on (doc hashes, policy version, sometimes a tenant-scoped config revision), so a semantically identical query against a shifted state is a miss by construction, not a threshold judgment call. That keeps the similarity threshold doing one job, paraphrase tolerance, instead of quietly moonlighting as a freshness check it was never designed for.
And to your last line: yeah, "high hit rate" without a paired false-hit measurement is one of those metrics that looks like ops maturity and is actually just confidence decay. Cheap to celebrate, expensive to audit later.