Embeddings and Vector Search in Practice

How you split a document decides what retrieval can ever find

Before anything gets embedded, it gets chunked, split into pieces small enough to embed and retrieve individually. Get the split wrong and no amount of tuning downstream fixes it, because the damage happened before the first vector was ever computed.

1def naive_chunk(text, size=500):
2    return [text[i:i + size] for i in range(0, len(text), size)]

This slices mid-sentence, mid-code-block, mid-table-row, wherever the 500th character happens to fall. A chunk that ends "the answer is 4" and starts the next chunk with "2, because..." embeds as two unrelated fragments, and retrieval has no way to know they were ever one thought.

The fix is splitting on the document's actual structure, paragraph breaks, headings, function boundaries in code, with a target size range instead of a hard cutoff, and a small overlap between adjacent chunks so a sentence spanning a boundary still appears whole in at least one of them.

Both directions can fail. Too small, and a chunk reading just "42" embeds with no context about what question it answers. Too large, and a chunk covering five different subtopics embeds as a vague average of all of them, scoring mediocre similarity against queries about any single one. The right size depends on the content, dense reference material wants smaller chunks than narrative prose, which is exactly why this step gets tuned empirically against real queries, not decided once and left alone.

Switching embedding models later means re-embedding everything

An embedding model turns a chunk of text into a vector, and the choice of which one to use trades off a few things worth deciding on purpose: higher dimension counts capture more nuance at the cost of storage and query time, and a model trained on general web text performs differently than one tuned for code, legal documents, or medical text, sometimes significantly.

The costly mistake isn't picking the wrong model, it's not realizing the decision is hard to reverse. Vectors from two different embedding models don't share a coordinate space, a query embedded with model A has no meaningful distance to a document embedded with model B. Switching models isn't a config change, it's re-embedding your entire corpus from scratch, which on a large document store is a real migration, not an afternoon's work.

Evaluate the embedding model against your actual queries and your actual documents before committing, the same way the chunking strategy above gets tuned empirically, not chosen off a benchmark leaderboard that may not resemble your content at all.

Comparing against every vector doesn't scale, so nobody actually does it

Finding the closest match to a query vector, exactly, means comparing it against every stored vector, one at a time. That's fine at a thousand documents and unusable at a hundred million, so production vector search doesn't do this at all, it uses an approximate nearest neighbor index instead.

1query_vector = embed(user_query)
2results = index.query(query_vector, top_k=5)
3for r in results:
4    print(r.score, r.chunk_id)

Structures like HNSW and IVF organize vectors so a query only has to check a small, promising fraction of the total, trading a small amount of recall, occasionally missing the true closest match in favor of the second- or third-closest, for a search that finishes in milliseconds instead of seconds.

That tradeoff is almost always worth it. A RAG pipeline retrieving the top five chunks out of a few hundred candidates that are all genuinely close rarely notices the difference between the true best match and the third-best one. The generation step downstream has a much bigger effect on answer quality than whether retrieval found the single mathematically closest vector.

Similar in meaning isn't the same as useful for answering the question

A query about "how do I enable two-factor authentication" and a document titled "why we don't recommend two-factor authentication for this feature" sit close together in vector space, they're about the same topic, using much of the same vocabulary. Semantically similar and actually useful for answering the question are different properties, and embedding similarity only measures the first one.

This is the gap that makes retrieval quality feel inconsistent even when every other piece is tuned well: good chunking, a well-chosen embedding model, a fast index, and the top result is still, on some fraction of queries, topically related but substantively wrong.

Vector similarity is a cheap first pass over a large corpus, not a precision instrument. Treating its output as the final answer instead of a shortlist is the mistake. The next article in this series covers reranking, a second, more precise pass over a much smaller candidate set, as the actual fix for this gap.