01

Short answer

Choose an embedding model by testing your languages, query patterns, corpus, chunk sizes, latency, and storage limits. Run candidates blind against real relevance judgments and release the winner through a shadow index.

Every retrieval-augmented system quietly inherits the quality of its embedding model. The vector database gets the attention and the architecture diagrams, but the model that produced the vectors determined most of what the system can find before a single query ran.

The choice is also surprisingly easy to get wrong, because the public leaderboards measure something different from what your corpus does to a model. Here is the practical process we work through in delivery.

02

Start from the workload, not the leaderboard

An embedding model maps text into a vector space where similar meanings sit close together. "Similar" is the load-bearing word. Models are trained with different notions of similarity, tuned for question-answer pairs, for passage-passage similarity, for code, for specific languages, and a model trained on English web text can quietly underperform on the mixture of English and Hindi that an Indian enterprise's documents actually contain.

Before looking at any benchmark, write down three things: the languages in the corpus, the query style (short keyword-ish questions, long descriptive requests, or full documents looking for neighbors), and the text lengths involved. These three constraints usually eliminate half the candidate list before any experiment runs.

Then look at leaderboard families like MTEB, with the right skepticism. They measure performance on public datasets with fixed definitions of relevance. Your corpus is private, your relevance judgments come from your domain, and a model two places lower on the board will sometimes beat the leader on your data by a margin that matters. The board tells you which models are in the conversation. Only your own evaluation tells you which one to ship.

03

The properties that matter in production

Dimensionality is the first practical cut. A model producing 3,072-dimensional vectors costs more to store and slower to search than a 768-dimensional one, at several million vectors the difference becomes real money and real milliseconds, and quality does not scale with dimension. Matryoshka-trained models let you truncate vectors to smaller sizes with graceful degradation, which is worth having when you do not yet know your scale.

Context length matters more than teams expect, because it decides whether a full document embeds as one vector or must be chunked before embedding. If the model handles your natural document length, chunking becomes a choice rather than a constraint, and fewer chunk boundaries means fewer retrieval edges.

Licensing and hosting decide your architecture. Open-weight models can serve on your own infrastructure, which matters for data-residency commitments and for cost at volume; hosted embedding APIs are simpler until the volume math turns. Some of the strongest open models in this space come out of the retrieval research community, and the trade between open weights and managed APIs is the same one we walk through for generation models in the inference cost guide.

Throughput and cost per token complete the picture. An embedding model runs over your whole corpus on every reindex, and over every query at serving time. A model that is 5 percent better and 10 times slower is a worse model for a system that reindexes nightly.

04

Running a bake-off that tells the truth

The evaluation is a few days of work and it is the only part of this that cannot be skipped.

Collect real queries, two hundred is a workable floor, from logs, from support tickets, or written by the people who actually answer these questions today. For each query, have someone who knows the domain mark which chunks of your corpus are relevant. This is the expensive step and the one that pays for everything else.

Then score each candidate model on that set. Recall at k is the metric that correlates with how the system will feel: of the relevant chunks, how many appear in the top ten the retriever returns. Average precision adds a sense of ordering. A model that wins on recall at k with the same latency and cost is the model you ship.

Two pitfalls repeat. Relevance judged by the person who picked the model poisons the comparison; use judges who did not see which model produced which ranking. And queries written by the engineering team, in clean grammatical form, do not resemble what users type; the bake-off must run on the ugly queries, typos and all, or it optimizes for a fiction.

05

Chunking is part of the embedding decision

The chunk size interacts with the model's training distribution and with your queries. Short chunks match short queries precisely but fragment long answers; long chunks carry context but dilute the vector when the relevant sentence is one line in five hundred words.

Run chunking variants as part of the bake-off rather than fixing chunks first and shopping for models second. Same evaluation set, same recall metric, and the chunking strategy that pairs best with the winning model is usually visible within one round of experiments.

For a deeper treatment of the retrieval side, our TF-IDF guide explains why a lexical baseline belongs in every one of these comparisons, and the RAG grounding guide covers what happens after retrieval: the answer layer that citations make trustworthy.

06

The maintenance question

Embedding models improve fast, and re-embedding a corpus to switch models has a real cost that scales with size. Decide up front how you will revisit the choice: re-run the bake-off against the incumbent every six months, on fresh queries, and switch only on a decisive margin. An incumbent that wins by a point is not worth a reindex; one that wins by ten on queries from last quarter's traffic is.

The teams that regret their embedding choice are rarely the ones that picked a mediocre model. They are the ones that never built the evaluation to notice the model had stopped being the right one.

07

Release through a shadow index

Build the candidate index beside the current one from the same version of the corpus. Keep chunking, filters, distance metric, and reranking fixed unless one of them is the variable under test. Changing several parts at once may improve the result without showing which change earned it.

Replay a representative query set against both indexes. Store the returned document IDs, ranks, scores, latency, and downstream answer result. Review disagreements rather than only average scores. Queries where both systems fail belong in the source or chunking backlog; queries where the candidate loses show the regression risk.

Then mirror a sample of live queries without serving the candidate result. Remove sensitive text from comparison logs, preserve the access filters applied to each request, and compare by language, topic, query length, and customer segment. The shadow period catches traffic patterns missing from the offline set while keeping users on the known path.

Promotion needs explicit gates. Set minimum retrieval quality by important slice, maximum p95 latency, storage limit, indexing time, and acceptable cost. Include zero-result and permission-filter behavior. A model that lifts average recall while losing the Hindi support slice should not pass because the aggregate score moved.

Switch traffic gradually if the serving layer allows it. Start with internal users or a small percentage, monitor retrieval disagreements and answer citations, then increase. Keep the old index until the observation period closes and rollback has been exercised.

Record the model version, tokenizer, dimensions, normalization, chunking version, index settings, and corpus snapshot used by the release. Those details make a later comparison reproducible and prevent a model name from standing in for the complete retrieval setup.

08

Related

Where to read next on the same subject.