The Two Numbers That Matter In Semantic Caching: Revision history

From Biblepay

Diff selection: Mark the radio buttons of the revisions to compare and hit enter or the button at the bottom.
Legend: (cur) = difference with latest revision, (prev) = difference with preceding revision, m = minor edit.

3 November 2026

  • curprev 18:2818:28, 3 November 2026 JuniorBrumbaugh talk contribs 4,624 bytes +4,624 Created page with "<br>Semantic caching is a practical way to cut the cost and latency of large-language-model applications. Instead of asking the model to answer every query from scratch, the system stores past question-answer pairs. When a new query arrives, its embedding is compared with the embeddings of stored queries. If the cosine similarity between the new query and a stored query is high enough, the cache returns the stored answer. The decision to reuse or recompute rests on a sin..."
Retrieved from "https://wiki.biblepay.org/The_Two_Numbers_That_Matter_In_Semantic_Caching"