Semantic search
Last verified: June 2026· definition
Semantic search uses embeddings to match a query to documents by meaning, so a search for “reset my password” can surface a doc titled “account recovery” even with no shared keywords.
It typically augments rather than fully replaces keyword search; hybrid approaches combine semantic similarity with traditional lexical matching and re-ranking for the best results.
Modernizing legacy keyword search with a semantic layer is a common entry point that often grows into a broader RAG or retrieval platform.
Hybrid retrieval is the production default
Pure semantic search has a predictable weakness: it is bad at exact tokens. Product codes, error strings, surnames, version numbers, and internal identifiers are precisely the things users search for and precisely the things embeddings blur, because a model trained on meaning has no reason to distinguish two similar-looking identifiers. Keyword search handles these perfectly. Running both and merging the result sets recovers the exact-match behaviour without giving up meaning-based recall, which is why almost every mature system does it.
Reranking usually beats a better embedding model
Retrieval has two stages, and teams tend to over-invest in the first. The initial vector search casts a wide net cheaply; a reranking model then scores each candidate directly against the query, which is far more accurate but too expensive to run over a whole corpus. Adding a reranker to an average retriever generally improves results more than replacing an average retriever with an excellent one, because it fixes ordering rather than just recall.
The evaluation problem
Semantic search fails in a way keyword search does not: it always returns something plausible. There is no empty result set to signal that nothing matched, so quality problems stay invisible until a user complains. The only reliable defence is a small labelled set of real queries with known-good results, checked whenever the retriever, chunking, or embedding model changes.
Filtering and permissions are part of retrieval
Most real systems cannot return every document to every user. Access control, tenant scoping, date ranges, and status filters all have to apply to search results, and where that filtering happens materially affects both correctness and performance. Filtering after retrieval can return a short or empty result set when the top matches were all inaccessible; filtering during retrieval avoids that but requires the metadata to live alongside the vectors, which is one of the stronger arguments for keeping vectors in the same database as your relational data.
Common misconceptions
MythSemantic search replaces keyword search.
RealityIt complements it. Semantic handles paraphrase and intent; keyword handles exact identifiers and rare terms. Systems that drop lexical matching usually regress on precisely the queries users care most about getting right.
MythIf results look reasonable, retrieval is working.
RealitySemantic search returns the nearest neighbours whether or not any of them are relevant, so plausible-looking output is the default rather than evidence of quality. Measuring against known-good answers is the only way to tell.
MythA better embedding model is the main lever.
RealityChunking, hybrid retrieval, and reranking typically move results more. Model choice matters, but it is rarely the constraint teams think it is.
Frequently asked questions
What is the difference between semantic search and RAG?
Semantic search is retrieval — it finds and returns relevant documents. RAG is a pattern built on retrieval, where the documents are then placed into a model prompt so it can generate a grounded answer. Semantic search is a component; RAG is a use of it. You can run semantic search with no model generating anything, and that is often the right product.
Why does semantic search miss exact matches like product codes?
Because embeddings encode meaning, and two similar identifiers have almost no meaningful distinction to encode. The vector for a specific code sits very close to the vectors for every other code of the same shape, so ranking between them is effectively arbitrary. Hybrid retrieval fixes this by letting keyword matching handle exact tokens while vector search handles everything else.
How do I know if my semantic search is any good?
Build a small evaluation set — a few dozen real queries with the results a knowledgeable person would expect — and measure how often the correct document appears in the top few results. It is unglamorous and it is the only thing that reliably catches regressions when you change chunking, swap an embedding model, or adjust how many results you retrieve.