How do AI engineers evaluate LLM and RAG systems in practice?

The evaluation architecture: three layers, one evidence trail

RAG is a system, not just a model call. A useful evaluation run preserves the case, the corpus and configuration that produced it, the retrieved items, the final answer, the citations, the scores, and the trace. That lets an engineer distinguish a retrieval failure from a generation failure instead of guessing.

flowchart LR
  C[Versioned evaluation case] --> R[Retrieval evaluation\nquery plus relevance labels]
  C --> G[Generation evaluation\nquestion plus gold evidence]
  R --> E[End-to-end evaluation\nreal retrieval plus real prompt]
  G --> E
  E --> P[Release gate and production monitoring]
  P --> F[Reviewed failures become new cases]
  F --> C

Microsoft’s current RAG guidance follows the same sequence: prepare test documents and queries, evaluate retrieval, then evaluate the answer using the retrieved grounding data. It names groundedness, completeness, utilization, relevance, and correctness as distinct measures (retrieval phase, end-to-end phase).

The diagram is not a mandate to use a particular vendor. It is a data contract. Every system should be able to answer, for each test case: what was asked, what evidence was available, what was retrieved, what was generated, how was it judged, and which version produced it?

Build a representative test set before selecting metrics

The central asset is not the evaluator library. It is a maintained set of questions whose expected behavior has been checked by people who understand the product and source material.

Use product data first, public benchmarks second

For a production application, assemble cases from sanitized query logs, support tickets, search failures, domain experts, documentation gaps, and usability research. Public benchmarks are valuable for early retrieval-model comparison and as a smoke test, but they do not establish that an internal knowledge assistant will answer an organization’s real questions correctly.

For example, BEIR is a heterogeneous benchmark for retrieval generalization, and MTEB offers retrieval tasks across domains and languages. They can help compare an embedding or reranking candidate before integration. RAGBench provides a large, industry-oriented RAG benchmark with explainable labels. None of these knows your document quality, permissions, terminology, recency requirements, or what a successful user outcome means.

Use this practical decision rule:

Start with a manually checkable pilot rather than a huge weakly labeled set. For example, a team might begin with 50 to 100 cases spanning the highest-value workflows, then expand from production failures. That is an illustrative starting point, not a universal sample-size rule. A safety-sensitive, multilingual, or high-volume product needs a larger and more deliberately sampled suite.

What one evaluation case should contain

Store cases in a portable format such as JSONL or a database table. At minimum, include:

• case_id, version, task description, user query, conversation context, and a clear expected behavior.
• A slice label, such as intent, customer segment, language, document type, ambiguity, freshness sensitivity, or risk level.
• answerable: yes, partially, no, or needs clarification. Negative and out-of-scope cases are first-class tests, not noise.
• Relevance labels, often called qrels, mapping the query to relevant document or chunk IDs. For multi-source answers, identify every required source and, where practical, the supporting span.
• A reference answer, a list of atomic facts that must be covered, or a domain-expert rubric. A single reference string is often too narrow for an open-ended answer.
• Expected citations, source-quality constraints, required format, refusal or escalation behavior, and any safety or policy boundary.
• The corpus snapshot identifier and the date the case was last verified.

Document-level labels are useful, but the retrieval unit must match the implementation. If the index retrieves chunks, label whether a candidate chunk contains the necessary support, not merely whether its parent document is relevant. Otherwise a chunker can appear to improve a metric while making the answer evidence unavailable.

Sample the cases by failure risk, not convenience

An eval set made entirely of short, obvious questions will produce an optimistic score and an unhelpful release gate. Deliberately include slices such as:

• Exact product names, abbreviations, typos, paraphrases, and multi-turn references such as “what about the one before that?”
• Single-document lookup, synthesis across documents, comparison, temporal questions, tables, images or OCR if relevant, and questions requiring a calculation or tool rather than retrieval.
• Contradictory, superseded, permission-restricted, and newly published documents.
• Ambiguous questions that should trigger clarification, plus no-answer and out-of-scope questions that should trigger a bounded response instead of invention.
• High-impact or compliance-sensitive topics, according to the product’s actual risk assessment.
• Languages, regions, and accessibility patterns actually represented in the expected user population.

Keep a development set for iteration and a locked holdout set for release decisions. Do not repeatedly tune against the holdout until it becomes a hidden development set. Record document and corpus versions so that a content refresh is not silently compared with an older index.

Evaluate retrieval as an information-retrieval problem

Retrieval evaluation asks a narrow question: did the ranked result list contain the evidence the answer needed? It does not ask whether the language model wrote a good answer.

Core retrieval metrics

For a query with known relevant items and a ranked list of results:

For a query requiring two supporting chunks, retrieved at ranks 1 and 3, Recall@3 = 2/2 = 1.00; if the returned list has three chunks and two are relevant, Precision@3 = 2/3 = 0.67; the first relevant result at rank 1 gives MRR = 1/1 = 1.00 for that case. Current Microsoft guidance describes Precision@K, Recall@K, and MRR for RAG retrieval and recommends evaluating both positive and negative examples (Microsoft Learn).

Choose K from the deployed architecture, not habit. If the retriever returns 20 results but the prompt builder supplies only the top 5, report both Recall@20 and Recall@5. More chunks can increase recall while increasing token cost, latency, distraction, and prompt-injection surface. Microsoft’s guidance specifically notes that too many chunks add noise and too few omit needed information (prompt design guidance).

Retrieval tests that uncover real causes

Run three complementary retrieval tests:

1. Label-based offline retrieval: compare top K IDs to qrels. This gives reproducible Recall@K, nDCG@K, and related metrics.
2. Oracle-context generation: bypass normal retrieval and give the generator the human-labeled evidence. If answers still fail, retrieval is not the only problem.
3. Negative retrieval: ask no-answer or deliberately out-of-domain questions. The system should not retrieve an attractive but unrelated chunk just to fill the context.

Use an error taxonomy as you inspect missed cases. Useful categories include:

This taxonomy prevents a common waste of time: repeatedly editing a prompt to solve a missing-document or parsing problem.

Evaluate answer generation with and without retrieval noise

Generation evaluation asks: when the needed evidence is present, does the application produce the right user-facing behavior? Use the same query with controlled contexts.

Test three evidence conditions

Explicit grounding instructions and defined fallback behavior are testable requirements. Current RAG prompt guidance recommends telling the model to use only provided context, cite each claim, acknowledge conflicts, and state when documents do not contain enough information (Microsoft Learn).

Measure more than “does it sound right?”

Citation support should be evaluated at the claim level, not by checking whether an answer merely contains links. A useful operational definition is:

• citation precision = supported cited claims / all cited claims
• citation recall = supported claims with at least one citation / all claims that require support

For example, one citation at the end of a five-sentence paragraph may look polished but does not establish which sentence it supports. Google’s grounding research evaluates both citation precision and citation recall and uses a claim-to-passage support check (Google Research).

Reference-free RAG metrics are useful for fast iteration when gold answers are scarce. RAGAS, for example, introduced metrics for context relevance, answer relevance, and faithfulness without requiring human ground-truth answers (RAGAS paper). Treat them as screening signals, not as final proof. A judge can miss a domain mistake, and a reference-free score cannot reveal that every retrieved source was out of date.

Use humans and model graders together, then calibrate the grader

Human review is slow and expensive. It is still necessary to define quality, resolve disputes, inspect consequential errors, and determine whether an automated grader matches the team’s actual standard.

A sensible division of labor

• Deterministic graders: JSON validity, exact calculations, schema fields, allowed citations, document permissions, a required refusal phrase, test API state, or a known expected value.
• Model graders: answer relevance, style, completeness against an atomic rubric, evidence support, and pairwise preference when several answers are reasonable.
• Human graders: ambiguous cases, high-impact tasks, new failure classes, model-versus-human disagreements, and calibration samples.

Anthropic’s practical evaluation guidance recommends deterministic graders where possible, model graders where necessary, and human review for validation. It also describes teams using product-defined criteria, periodic human calibration, and separate quality and regression suites (Anthropic). OpenAI likewise uses expert blind comparisons and detailed rubrics for real-world tasks, while stating that its automated grader is not used to replace expert review (OpenAI GDPval).

Calibrate a model grader before letting it gate a release

1. Write a rubric with observable criteria. Prefer “does each stated policy condition have support?” over “is the answer excellent?”
2. Create a held-out calibration sample with human labels, including borderline answers, correct abstentions, and known bad examples.
3. Compare the model grader to humans. Report agreement and, for an important failure label, false positives and false negatives. Set a threshold from the product’s risk tolerance, not from a default score such as 3 out of 5.
4. For A/B judgments, blind the candidate identity and reverse answer order on a sample. Keep the question, evidence, and rubric constant.
5. Send low-confidence, contradictory, or high-impact judgments to humans. Periodically refresh the calibration set as the product and corpus change.

LLM judges can scale feedback, but they are not neutral instruments. Research on MT-Bench and Chatbot Arena found position, verbosity, and self-enhancement biases, even while showing substantial agreement with humans in its studied setting (Zheng et al., 2023). The practical response is not to abandon model grading. It is to use a clear rubric, check the grader against human decisions, randomize comparisons, retain disagreement cases, and never treat one uncalibrated judge score as ground truth.

Slice analysis turns a score into an engineering decision

An average can conceal the failure the product cannot afford. Always show the overall score and a breakdown by the conditions that could change the decision.

Useful slices include query intent, document type, source freshness, language, user role and permissions, query length, multi-hop versus single-hop questions, answerable versus no-answer cases, high-risk topics, and source system. For each slice, report its number of cases, the primary metric, change from baseline, and a confidence interval or uncertainty estimate where the test set is small.

Consider a retriever with 0.90 overall Recall@5. If it is 0.98 for short product-name lookup but 0.42 for questions requiring two documents, the next change should target multi-document retrieval, not declare the system ready. Likewise, a response with high groundedness but low completeness may be using its retrieved context faithfully while omitting a required condition. Microsoft’s end-to-end guidance gives the same diagnostic example: groundedness and correctness, or utilization and completeness, must be read together rather than as a single score (Microsoft Learn).

Make release gates explicit and repeatable

A regression gate says what must remain true to ship any system change, whether the change affects documents, parsing, chunking, embeddings, retrieval, reranking, prompt templates, tool behavior, model settings, or UI.

Record the full evaluation manifest

For every candidate and baseline run, record:

• Test-set version, rubric version, corpus snapshot or content hashes, and document access policy.
• Ingestion/parser version, chunking strategy, embedding model, index configuration, query rewrite, filters, retriever, reranker, and top K.
• Prompt template, examples, model identifier, decoding settings, tool definitions, and safety settings.
• Retrieved IDs and scores, final answer, citations, full trace where relevant, evaluation scores and reasons.
• Token counts, model and retrieval latency, error status, and estimated cost.

Without this manifest, a score change is not reliably attributable. With it, an engineer can replay a failure and test a targeted hypothesis.

Use a gate with floors, not a single composite score

There is no universal pass number. Define thresholds from the harm of each failure and the value of the task. A low-risk internal brainstorming assistant can tolerate a different error profile from a customer-support or regulated workflow.

Run the baseline and candidate against the same fixed cases and corpus snapshot. For stochastic models, run enough repeated trials on known-unstable or consequential cases to see variation instead of treating one response as definitive. Change one meaningful variable at a time when diagnosing an improvement. Record the rationale, as current RAG prompt guidance recommends for prompt versions and their evaluation results (Microsoft Learn).

A small worked RAG evaluation example

This is a hypothetical support assistant, not a claim about a real company or policy.

Task: Answer “Can I cancel my annual plan after 10 days and receive a refund?”

Gold evidence labels:

Retriever output: [A-annual-terms-4, C-old-policy-1, B-refund-exclusions-2]

• Recall@3 = 2/2 = 1.00, because both required chunks appeared.
• Precision@3 = 2/3 = 0.67, because the obsolete chunk is noise.
• MRR = 1.00, because the first result is relevant.
• Recall@2 = 1/2 = 0.50, which matters if the prompt builder sends only two chunks to the model.

The results say that retrieval at K=3 is adequate for this single test, but context selection at K=2 is not. They do not say the answer is correct.

Candidate answer: “You can request a refund when cancelling within 14 days, but it is subject to the listed eligibility condition. [A] [B]”

The answer reviewer checks two atomic claims:

For this answer, citation precision is 2/2 and citation recall is 2/2; the expected policy conditions are both covered, so the completeness rubric passes. An end-to-end release gate would still check the same behavior across the relevant corpus, test the no-answer version of the question, inspect stale-policy handling, and confirm p95 latency and cost stay within budget. One passing example is a debugging aid, not evidence that the system is ready.

Monitor the deployed product, not just the offline suite

Offline evaluation determines whether to try a change. Production monitoring determines whether assumptions still hold. NIST calls for monitoring AI-system functionality and behavior in production and for tracking unanticipated risks over time (NIST AI RMF).

Monitor at least four kinds of signals:

Online experiments should start with a bounded audience or shadow traffic when the risk permits. Do not quietly expose a consequential workflow to an untested candidate merely because its offline average rose. A product decision should be based on user outcome and risk, not on a leader-board score alone.

Cost and latency belong in the quality decision

For RAG, quality has a budget. Increasing K, adding a reranker, requesting more model reasoning, or using a second judge can improve one offline score while pushing a workflow past its latency or unit-cost target.

Measure per case and by slice:

• Retrieval latency, reranking latency, model time, and end-to-end p50 and p95 latency.
• Input and output tokens, API or compute cost, number of retrieval calls, cache behavior, retries, and error rate.
• Cost and latency per successful or safely handled task, not only per request. A very cheap answer that causes repeat searches or escalation is not necessarily cheaper for the product.

Compare candidates on a Pareto frontier: a change is compelling when it improves task success or a critical risk measure without violating a latency and cost constraint. If a new reranker adds 600 ms but prevents failures on a costly, high-value slice, it may be worth it. If it slightly improves aggregate relevance but harms p95 latency for every user, it may not be.

Common failure modes and better alternatives

A practical implementation checklist

1. Write down the one or two user tasks the application must help complete and the harm of getting each wrong.
2. Collect representative questions and label answerability, relevant source IDs, required facts, and expected behavior.
3. Establish retrieval baselines with Recall@K, Precision@K, and nDCG@K or MRR where they fit the task.
4. Run oracle-context tests to separate retrieval gaps from prompt or generation gaps.
5. Add claim-level groundedness, citation support, completeness, correctness, format, and no-answer checks.
6. Create a short human-labeled calibration set. Validate every model grader against it before using the grader in a release decision.
7. Tag each case with meaningful slices and inspect the worst slices, not only the average.
8. Store an evaluation manifest and run the suite in CI or another repeatable release process.
9. Define explicit quality, safety, p95 latency, and cost gates. Document exceptions and their expiry date.
10. Monitor production, review sampled failures, and convert durable new failure modes into versioned offline cases.

Tooling is replaceable, the evidence trail is not

Teams may implement this with a custom JSONL runner and CI, an open-source evaluation library, an observability platform, or a managed evaluation service. The original question mentions RAGAS, DeepEval, LangSmith, and TruLens. They can reduce implementation work, but no public source can establish that any one of them is the industry standard for every product. Choose tooling that can preserve your data contract: replayable cases, corpus and configuration versions, retrieval traces, grader prompts and versions, raw outputs, human labels, slice reports, and exportable results.

For a small team, a simple, well-versioned harness plus human review of failures is often more valuable than an elaborate dashboard with uncalibrated default metrics. For a larger system, a platform that manages traces, datasets, dashboards, and reviewer queues can be worthwhile. Either way, a tool should make the evaluation program more reproducible, not obscure it.

Limits and boundaries

No finite benchmark proves that an LLM or RAG system is reliable for every future question. Offline metrics are estimates over a sample, user feedback is selective, and corpus, model, and traffic distributions change. Benchmarks can also be contaminated or become familiar to models, so use them as evidence, not as a certification.

High-stakes legal, medical, financial, employment, safety, or security workflows need domain-specific review, escalation, access control, privacy controls, and risk thresholds. This article is engineering guidance, not legal, medical, financial, security, or compliance advice. Do not use an uncalibrated model judge as the sole control for a consequential decision.