Semantic caching

An exact-match cache only helps when two requests are byte-identical. A semantic cache embeds the incoming request and looks for a previous request whose meaning is close enough, returning the stored answer instead of paying for a fresh generation.

It shines on workloads with a hot set of common questions — support assistants, docs Q&A, internal helpers — where users ask the same thing a hundred different ways. Hit rates of 20–40% are realistic on that shape of traffic, and every hit is a request you did not pay for and did not wait for.

The risk is a false hit: returning a cached answer for a question that only looked similar. That is a tuning problem (similarity threshold, namespacing by user or tenant, TTLs on volatile data) and, like every cost optimization, it should sit behind an eval so you can prove quality held.