How to build semantic RAG AI search

Semantic search with embeddings finds text by meaning, not by matching exact words. You turn each document into a vector, which is a long list of numbers that captures what the text is about. You store those vectors. When a query arrives, you embed it the same way and compare it against every stored vector. The closest ones come back as your results. The comparison is a small piece of math called cosine similarity. That is the whole idea, and it is most of what I built for retrieval at Carpathian.

Keyword search asks which documents contain these words. Semantic search asks which documents mean the same thing. Someone can type "my card was declined" and match a help page titled "payment failed at checkout," even though the two share no words. This guide covers what an embedding is, how the ranking works, how to index your own content, and where the approach falls short. By the end you can build a small version yourself.

Weak search costs money. A Google Cloud-commissioned Harris Poll survey of nearly 13,500 shoppers found that search abandonment costs retailers more than $2 trillion a year worldwide, and that 76% of US shoppers ran a search that did not return the item they wanted (Google Cloud). Matching on meaning is one way to close that gap.

What is an embedding?

An embedding is a list of numbers that stands in for a piece of text and captures its meaning. A model reads the text and prints out the numbers. Text with a similar meaning gets similar numbers, so it lands in a similar place. The numbers themselves are not readable. You never look at them by hand. What matters is that meaning turns into position, and position is something a computer can compare fast. Each number is one coordinate. Put enough coordinates together and you have a point in a very high-dimensional space. Two texts that mean the same thing sit close together in that space. Two texts about different topics sit far apart. That single property is what makes search by meaning possible at all.

How does semantic search with embeddings rank results?

Semantic search with embeddings ranks results by how close each stored vector sits to the query vector, using cosine similarity. First you embed the query into the same kind of vector as your documents. Then you score every stored vector against it. The score is a number between minus one and one. Higher means more alike. You sort by that score, high to low, and keep the top few. In my retrieval code the default is the closest twelve chunks. Those go to the model or straight back to the user. There is no keyword step in the middle. The vectors do all the matching, so a result can rank first even when it shares no words with the query.

Cosine similarity sounds fancier than it is. It measures the angle between two vectors and ignores their length. Point the same direction, and the score is one. Point at right angles, and the score is zero. In plain terms, it asks whether two pieces of text lean the same way in meaning, not how long either one is. My version walks every stored chunk, does a little arithmetic per pair, and sorts the results. That is a brute-force scan. It is exact, easy to reason about, and quick when you have thousands of chunks. When you reach millions, you swap the scan for an approximate nearest-neighbor index, which trades a sliver of accuracy for a large jump in speed.

How do you turn your documents into a searchable index?

You break each document into chunks, embed every chunk, and store the vectors next to the text they came from. A whole PDF is too big to embed as one unit, so you split it. In my indexer I aim for chunks of about 1,800 characters, cut on paragraph boundaries where I can, with a 250-character overlap between neighbors. The overlap matters. An answer that straddles the seam between two chunks would otherwise get sliced in half and lost. Each chunk gets embedded and saved with its page numbers, so a match can cite where it came from. For scanned or image-only pages, the text gets pulled out with OCR first, then chunked and embedded like anything else.

Keeping the index fresh is the part people forget. An embedding describes the text as it was on the day you indexed it. Change the document and the old vector is now wrong. When someone edits a page in my system, I re-embed only that document rather than rebuilding everything, which keeps the cost small. A background sweep also retries anything that stalled, so a single failed upload does not sit broken forever. The rule to carry with you: if content can change, you need a plan to re-index it, or your search quietly drifts out of date while looking perfectly healthy.

What else can you build with embeddings?

Once you can embed text and rank by similarity, you have a single primitive that does four different jobs. Find-similar takes one item and returns its nearest neighbors. Recommendations do the same thing from a user's history instead of a single item. Deduplication flags two records whose vectors sit almost on top of each other. Retrieval-augmented generation, or RAG, pulls the closest chunks and hands them to a language model as context. The mechanics underneath are identical every time: embed, compare, rank. Only the input and what you do with the output change. Learn the primitive once and you get all four.

This is why the same step keeps turning up across features that look unrelated on the surface. The retrieval I built for grounded question answering is the same embed-and-rank move you would reach for to power a "more like this" widget. It also shows up when you score content for quality or moderate what users submit, where you compare new text against known examples. If you have built one of these, you are most of the way to the others.

Why run inference through an OpenAI-compatible endpoint?

An OpenAI-compatible endpoint lets you swap the model behind your search without rewriting your app. It is a shared request-and-response shape that many tools already speak. You send messages to a path like /v1/chat/completions, and you get back a standard reply with the answer and a token count. Your code targets that shape, not a specific vendor. On Carpathian's managed inference, the API key you send picks the model, so pointing at a different one is a settings change rather than a code change. The embedding side speaks the same language through /v1/embeddings, so both halves of your search stack share one contract.

The payoff is freedom to change your mind later. A better or cheaper model shows up, and you move to it without touching the app that calls it. My proxy even normalizes replies from a local engine back into the standard shape, so the caller cannot tell the difference. That is how Veritate, our open model work, plugs into the same interface a hosted model would use. Pick the compatible surface early. It costs nothing today, and it saves you a painful rewrite the first time you want to switch models, which you will.

When should you still use keyword search?

You should still use keyword search whenever the query is an exact string that has to match exactly. Embeddings are built to blur near-meanings together, and that is precisely the wrong behavior for an order number, a SKU, an error code, or a person's last name. Search for ticket "INC-4471" and a meaning-based system may happily return "INC-4470," because the two look almost the same to it. That is not a bug you can tune away. It is what matching on meaning does. For identifiers and precise terms, plain keyword lookup is faster, cheaper, and correct.

The fix most teams land on is hybrid search: run both, then blend the scores. Keyword catches the exact hits, embeddings catch the fuzzy ones, and together they cover far more queries than either alone. Two more honest limits are worth stating plainly. Embeddings cost compute to create, so re-indexing a large, fast-changing corpus adds up. And a stale index looks fine while quietly returning yesterday's answers, which is worse than an obvious error because nobody notices.

Embeddings get you most of the way to reliable search without relying on ctl-f searches, with a surprisingly small amount of code. Keep the exact-match cases on keyword search, keep your index fresh, and let meaning do the rest. If you would rather have this built and tuned for your own data, that is the kind of work our team takes on through our AI application development practice.