Azure AI Search for RAG: The Retrieval Decisions That Actually Decide Answer Quality

When a retrieval-augmented generation system gives a wrong or evasive answer, the first instinct is almost always to rewrite the prompt. It's the visible part, it's the cheapest to change, and every so often it genuinely helps — which is precisely why it becomes a habit.

Most of the time, though, the prompt was never the problem. If the passage that actually contains the answer never made it into the context window, no amount of prompt tuning recovers it. The model was handed a question and a pile of text that doesn't answer it, and it did what language models do with an underspecified task: it produced something confident and plausible anyway.

That failure happens upstream, in retrieval. And in Azure AI Search, retrieval quality comes down to four decisions that matter far more than which model generates the final answer: how you chunk, how you search, how you filter, and how you rank. This piece works through each one, with a query shape you can adapt directly, and closes with the two features — evaluation and agentic retrieval — that separate a system someone tuned once from one that keeps working.

Chunking sets the ceiling before embedding even starts

Every document has to be split into chunks before it's embedded, and the split you choose puts a hard ceiling on what can ever be retrieved later. No amount of clever ranking downstream fixes a bad chunk boundary.

Get the size wrong in either direction and retrieval suffers in a different way:

• Chunks that are too large cover several topics at once. Their embedding becomes an average of everything inside them, which matches nothing precisely, and they crowd out other candidates when the model's context budget is fixed.

• Chunks that are too small lose the surrounding context that made them meaningful. A step that says "then click Save" is useless retrieved on its own, with no idea which procedure it belongs to.

Two practices fix most of this without much extra engineering. First, split on structure, not on a fixed character count — headings, sections, and paragraph boundaries carry meaning that a naive fixed-size window destroys, particularly when it cuts a table in half. Second, carry the heading path into the chunk itself: prefixing a chunk with its document title and section heading measurably improves matching, because the chunk now states what it's about instead of relying on the reader to infer it. A small amount of overlap between adjacent chunks is also worth the redundancy — it stops an answer that straddles a boundary from being lost by both neighboring chunks.

Azure AI Search's integrated vectorization pipeline gives you three chunking skills with different trade-offs, and picking the wrong one for the document type is a common source of quiet retrieval failures:

Hybrid search, not vector search alone

Pure vector search is excellent at meaning and weak at exactness. Ask about an error code like 0x8007000E or a part number, and the embedding has almost nothing to grab onto — those strings carry no semantic content, and a near-miss code can sit right next to the correct one in vector space. Keyword search is the mirror image: precise on rare, literal terms and useless the moment a user paraphrases instead of quoting.

Hybrid search runs both in parallel — BM25 keyword ranking and vector similarity — against the same query, then fuses the two ranked lists using Reciprocal Rank Fusion (RRF), an algorithm that combines rankings based on each result's position in every list rather than trying to reconcile two different scoring scales directly. Because real question sets are always mixed — people ask conceptual questions and quote exact error strings, often in the same sentence — hybrid search beats either approach alone by a wide margin on realistic workloads. Treat hybrid as the default and reach for vector-only search only after you've measured that it actually wins on your own data.

A hybrid request combines both parameters in one call:

json

POST https://{search-service}.search.windows.net/indexes/{index}/docs/search?api-version=2026-04-01
{
  "search": "how long is the notice period for a fixed-term contract",
  "vectorQueries": [
    {
      "kind": "text",
      "text": "how long is the notice period for a fixed-term contract",
      "fields": "contentVector",
      "k": 50
    }
  ],
  "queryType": "semantic"
}

Using "kind": "text" instead of supplying a raw embedding array lets Azure AI Search vectorize the query itself at request time, through the vectorizer configured on the field — the query-side half of integrated vectorization. Set k generously (50 is a reasonable default) when you're feeding results into the semantic ranker next, since the ranker needs a wide enough candidate pool to have something worth reordering.

Filters are correctness, not tuning

It's tempting to file filters under performance optimization. They aren't — they're what stops retrieval from returning material the user shouldn't see, or material that's no longer true. Get this wrong and the system doesn't fail loudly; it fails by confidently stating something false or something someone else's data.

Security trimming is the most consequential case. Store the identities — group IDs, user IDs, or Entra object IDs — permitted to see each document as a filterable, non-retrievable collection field, and filter on the caller's group membership inside the query itself:

json

{
  "name": "group_ids",
  "type": "Collection(Edm.String)",
  "filterable": true,
  "retrievable": false
}

json

{
  "filter": "group_ids/any(g: search.in(g, 'hr-all,employees'))"
}

The search.in() function is deliberately built for this: it stays fast even against hundreds or thousands of group IDs, where an equivalent chain of or comparisons would not. This has to run inside the retrieval query, not after — filtering after retrieval means the disallowed material was already loaded into context, and post-filtering an already-empty result set produces an answer built from nothing at all. (Azure AI Search also offers built-in document-level access control lists as an alternative to hand-rolled group filtering, worth evaluating if your authorization model maps cleanly onto Entra ID groups.)

Two other filter categories deserve the same discipline:

• Recency and validity. Superseded content is the quiet killer of RAG quality — nothing errors, the system simply states the old policy with total confidence. Store an effective date or a status field and filter on it.

• Scope. Product, region, language, business unit. Narrowing the candidate set before ranking improves both precision and latency, since the ranker has fewer irrelevant candidates to sort through.

Filters run before ranking and cost very little to evaluate, which makes them one of the highest-leverage changes available — they remove noise the rest of the pipeline would otherwise have to work around.

Semantic ranking: cheap quality on top of retrieval

Vector and keyword retrieval return candidates in similarity order, which is a reasonable but rough proxy for actual relevance. Semantic ranking re-scores the top results — up to the top 50 — using Microsoft's semantic models, adapted from the same language-understanding work behind Bing, which read the query and each passage together rather than comparing them as isolated embeddings.

Each document is scored on a 0–4 scale (@search.rerankerScore), built from up to roughly 2,000 tokens extracted from the title, keyword, and content fields, in that priority order. A score near 4 means the passage essentially answers the query outright; near 0 means it's unrelated despite having made the initial candidate list. Beyond reordering, semantic ranking can also return extractive captions with key terms highlighted, and optional semantic answers for direct question-style queries — both useful for showing a reader why a result was returned, not just that it was.

Semantic ranking is billed separately from your search tier — there's a free monthly allowance, with standard consumption pricing beyond it, and it's only charged when queryType=semantic is combined with an actual (non-wildcard) search string. It's also only available in select Azure regions, worth checking before you plan around it.

Ordering matters more than intuition suggests. Language models attend unevenly across a long context window, and the single most relevant passage sitting fifth instead of first can measurably change the generated answer. If you're passing five chunks to the model, which five — and in what order — is a bigger lever on answer quality than most prompt rewrites will ever be.

Putting it together: a query shape that works

The four decisions above compose into a single request. Retrieve broadly with hybrid search, restrict to what the caller is allowed to see and what's still current, re-rank semantically, and return only what the generation step actually needs:

json

POST https://{search-service}.search.windows.net/indexes/{index}/docs/search?api-version=2026-04-01
{
  "search": "how long is the notice period for a fixed-term contract",
  "vectorQueries": [
    {
      "kind": "text",
      "text": "how long is the notice period for a fixed-term contract",
      "fields": "contentVector",
      "k": 50
    }
  ],
  "queryType": "semantic",
  "semanticConfiguration": "default",
  "filter": "status eq 'current' and group_ids/any(g: search.in(g, 'hr-all,employees'))",
  "select": "id,title,section,content,sourceUrl,effectiveDate",
  "top": 5
}

Read in order: retrieve fifty candidates by both keyword and vector matching, restrict to current documents the caller may read, re-rank the survivors semantically, and return the best five with the metadata needed to cite them.

Evaluate, or you're guessing

Every real improvement to a retrieval pipeline comes from measurement, and the measurement itself isn't complicated:

1. Collect fifty to a hundred real questions — pulled from an actual support queue or usage log, not invented by the team building the system.

2. For each one, record which document actually contains the answer.

3. Measure whether that document shows up in the top k retrieved chunks (this is recall@k, the single most useful number in this whole exercise).

4. Change one variable at a time. Measure again.

That one number — did retrieval surface the right source at all — is what separates a retrieval problem from a generation problem. Without it, every change is a guess dressed up as an improvement.

Beyond a single query: agentic retrieval for multi-part questions

Everything above assumes one query can represent what the user is asking. Often it can't. "Find a policy that covers both remote work and international contractors, and tell me if it's changed since last year" isn't one lookup — it's several, and a single hybrid query tends to average across the sub-questions rather than answering any of them well.

Agentic retrieval, now generally available through the Azure AI Search REST API, addresses this by putting a query-planning step in front of retrieval. An LLM breaks a complex question — and, where relevant, the surrounding conversation history — into smaller, focused subqueries, which then run in parallel across your configured knowledge sources (an indexed source backed by a search index, or a remote source fetched at query time). Each subquery's results go through the same semantic reranking described above, and the results are synthesized into a single response with source references.

It adds latency and cost — you're now paying for retrieval tokens, semantic ranking, and an LLM's query-planning and synthesis calls — so it isn't a default replacement for the query shape earlier in this article. It's worth reaching for specifically when your question set includes genuinely multi-part questions, ambiguous phrasing that benefits from query rewriting, or conversational follow-ups that depend on earlier turns. As of this writing, the REST API path (api-version=2026-04-01) is GA; the Azure portal and Microsoft Foundry portal experiences for configuring it are still in preview, so expect to wire this up through code or infrastructure-as-tooling rather than a portal wizard for now.

Common mistakes worth checking first

• Blaming the model for a retrieval failure. Check what was actually retrieved before touching the prompt.

• Fixed-size chunking on structured documents. Split on headings and layout, not a raw character count.

• Vector-only search. Error codes, part numbers, and product identifiers need keyword matching alongside it.

• Filtering after retrieval instead of inside the query. Security trimming has to happen at query time.

• Indexing everything, including superseded content. Stale documents produce confident wrong answers, not blank ones — they're worse than a gap.

• No citations in the final answer. Without a source, a reader can't verify anything and will either over-trust the system or abandon it.

• Passing twenty chunks because the context window is large. More context isn't more accuracy; it dilutes the model's attention and raises cost for no benefit.

Frequently asked questions

How many chunks should I actually pass to the model? Usually three to five well-chosen chunks outperform twenty mediocre ones. Retrieve a wide candidate set, re-rank it, then pass only the top few. Measure this against your own question set rather than adopting a number from an article — including this one.

Does a bigger or newer embedding model fix poor retrieval? Rarely on its own. Chunking quality and hybrid search almost always dominate embedding model choice. Change the embedding model last, and only after you have a recall@k measurement showing whether it actually helped.

How do I keep the index current as source documents change? Indexers can pull from supported data sources on a schedule with built-in change tracking. For sources indexers don't cover, push updates as documents change instead. Either way, decide explicitly what happens to deleted source documents — an index still full of deleted content reproduces the exact stale-answer problem described above.

Can I do RAG without a vector index at all? Yes. For a small, well-titled corpus, keyword search combined with semantic ranking can work well on its own. Vectors earn their place once users start phrasing questions differently from how the source documents are written — which, at any real scale, is most of the time.

How do I stop the system from answering when nothing relevant was actually found? Apply a score threshold, and when nothing clears it, return an explicit "not found" rather than passing weak matches through to the model anyway. A model handed irrelevant context will still generate a fluent-sounding answer, and that confidently wrong answer is the worst output the system can produce.

Where to spend the next day

If a RAG system is underperforming, spend the time on retrieval, not the prompt. Build a real question set, measure whether the right source is actually being found, fix chunking on structured content, turn on hybrid search and semantic ranking, and add the filters that remove content that should never have been a candidate in the first place. Generation quality tends to take care of itself once the model is finally given the right paragraph to work with.