The Two Numbers That Matter In Semantic Caching
Semantic caching is a practical way to cut the cost and latency of large-language-model applications. Instead of asking the model to answer every query from scratch, the system stores past question-answer pairs. When a new query arrives, its embedding is compared with the embeddings of stored queries. If the cosine similarity between the new query and a stored query is high enough, the cache returns the stored answer. The decision to reuse or recompute rests on a single similarity threshold, typically shipped as a default between 0.80 and 0.95.
That threshold is the whole game. Set it too low, and the cache returns answers for questions that only look similar but are actually different. Set it too high, and the cache refuses to reuse answers even for queries that are genuinely the same. Engineers usually tune this number by eyeballing a few examples. A 2026 preprint by Guerin Green of Novel Cognition research Cognition argues that this approach hides the real problem: you need to measure false matches and missed matches as two separate numbers, not as one blended accuracy score.
Consider a customer who asks, "How do I cancel my subscription?" versus "How do I pause my subscription?" The words are close, the word order is close, and the cosine similarity of their embeddings will be high. But the answers are different. A semantic cache that reuses the cancel answer for the pause query produces a false cache hit: the user gets a confident but wrong response. The opposite failure is a missed match: a user asks, "What is the refund policy for annual plans?" and then later asks, "Could you tell me how refunds work for yearly subscriptions?" These are the same question, but if the embeddings land just below the threshold, the cache misses and the model recomputes an identical answer.
The preprint built a labelled set of 243 English customer-support question pairs across eight domains. It contained 101 heavily reworded paraphrases, 102 near-miss pairs that differ by a word or by word order but need different answers, and 40 lightly reworded paraphrases reported separately. Using the open embedding model Qwen3-Embedding-0.6B, the researchers scored every pair by cosine similarity. The result was striking: the near-miss pairs scored higher on average, with a median of 0.801, while the genuine paraphrases scored lower, with a median of 0.691. The area under the ROC curve was 0.144, meaning the two distributions were inverted, not merely overlapping. The embedding model was more likely to judge a near-miss as similar than a true paraphrase.
At a threshold of 0.85, which is the default of one commercial service, that model matched none of the 101 paraphrases and 29 of the 102 near-misses, a 28.4 percent false match rate. To get false matches down to zero, the threshold had to be raised above 0.985, but at that level 2 of the 40 near-verbatim repeats were still missed. There is no single threshold that works for both numbers. The preprint also found that similarity correlated with word-set Jaccard overlap, with a Pearson r of 0.695, even within each label. Adding a query-instruction prefix halved false matches at 0.85, dropping them to 12.7 percent, without changing the ordering of the pairs.
The core argument is that a similarity threshold is not a universal constant. It is a property of one embedding model on one traffic distribution. A threshold that works for Qwen3-Embedding-0.6B may be useless for another model, and a threshold tuned on one set of customer queries may fail on another. Other embedding models need their own measurement. The preprint describes a labelled near-miss protocol for setting the threshold, and it releases the pairs, the scores, and an analysis script under CC BY 4.0 so that anyone can recompute every statistic.
For engineers, the practical takeaway is to track false cache hits and missed matches separately. A single accuracy number hides the trade-off. If you only look at overall hit rate, you will not see that your cache is confidently answering the wrong questions. If you only look at false matches, you will raise the threshold until your cache is nearly useless. The right approach is to build a small labelled set of near-miss queries and genuine paraphrases, measure both numbers across a range of cosine similarity thresholds, and then choose the threshold that balances the cost of a wrong answer against the cost of a recompute. That is a product decision, not a model property.
