Semantic Caching Is Not A Threshold Problem
Semantic caching promises a simple win: store a large-language-model response, and when a new query arrives, compare its embedding to the stored query’s embedding. If the cosine similarity is high enough, return the old answer instead of calling the model again. In deployed products, that decision is governed by a single similarity threshold, typically shipped as a default between 0.80 and 0.95. The intuition is that anything above that range is a paraphrase, and anything below is a new question. That intuition is wrong in a specific, measurable way, https://zenodo.org/records/23667256 and the defaults are not a safe place to start.
Consider two customer requests: "cancel my subscription" and "pause my subscription." They share most words, and their embeddings will be close. But the answers are different: one ends a service, the other suspends it. A semantic cache that treats those as equivalent produces a false cache hit—a wrong answer delivered with confidence. Conversely, "I’d like to stop the monthly plan" and "please terminate my recurring billing" are genuine paraphrases, but their embeddings may be further apart than the near-miss pair. A threshold that catches the near-miss will miss the paraphrase; a threshold that catches the paraphrase will accept the near-miss. This is not a tuning problem. It is a structural property of how embeddings represent meaning.
A 2026 preprint by Guerin Green of Novel Cognition tested this directly. The author built a labelled set of 243 English customer-support question pairs across eight domains: 101 heavily reworded paraphrases, 102 near-miss pairs that differ by a word or by word order but need different answers, and 40 lightly reworded paraphrases reported separately. Scored by cosine similarity with the open embedding model Qwen3-Embedding-0.6B, the near-miss pairs scored higher (median 0.801) than the genuine paraphrases (median 0.691). The area under the ROC curve was 0.144, meaning the two distributions were inverted: near-misses looked more similar than actual paraphrases. At a threshold of 0.85—the default of one commercial service—that model matched none of the 101 paraphrases and 29 of the 102 near-misses, a 28.4% false cache hit rate. False matches reached zero only above a threshold of 0.985, where 2 of the 40 near-verbatim repeats were still missed.
The problem is not just the numbers; it is the assumption that a threshold is a universal constant. Similarity correlated strongly with word-set overlap (Pearson r = 0.695), meaning the model was largely measuring lexical closeness, not semantic equivalence. A query with a small word change—like swapping "cancel" for "pause"—gets a high score even though the meaning shifts. Adding a query-instruction prefix halved false matches at 0.85 (to 12.7%) without changing the ordering of the pairs, but that is a workaround, not a fix. The threshold is a property of one embedding model on one traffic distribution. It does not transfer.
What practitioners should take from this is that cache hit rate—the percentage of incoming queries served from cache—is not the metric that matters. A high cache hit rate with a low threshold is easy to achieve and dangerous. The real question is how many of those hits are false. A better approach is to build a labelled set of near-miss pairs from your own traffic, measure where your embedding model places them relative to genuine paraphrases, and set the threshold above the highest near-miss score—not at a vendor default. The preprint describes exactly such a protocol and releases the pairs, scores, and analysis script under a permissive license. Other embedding models need their own measurement; there is no shortcut. Semantic caching works only when you know what your embedding model actually considers similar, and that knowledge requires labelled data, not a default.
