Production RAG Evaluation and Knowledge Governance
Separate retrieval and generation failures, govern source versions, review evidence, and control production RAG costs without treating architecture patterns as product guarantees.
A production RAG assistant can retrieve the right document and still give the wrong answer. It can also quote an outdated policy faithfully. Those failures need different fixes, and a single answer-quality score hides the distinction.
Production operations need a repeatable evaluation set, a maintained knowledge lifecycle, and a record of why evidence was accepted or rejected. The architecture patterns below are engineering recommendations. They do not imply that Tencent Cloud Agent Development Platform (ADP) has shipped every component described here.
Separate retrieval failures from answer failures
Start with a versioned evaluation set built from actual tasks and reviewed failure cases. Include direct lookups, cross-document questions, follow-up questions, missing information, conflicting versions, and access-denied requests. Record the user role, expected source passages, applicable date, and acceptable answer or refusal. Keep a held-out set for checking changes that were tuned on development examples.
For an illustrative policy question, ask whether a service request is eligible under the current rules. A failure to retrieve the current eligibility clause is a retrieval problem. An answer that ignores an exception in the retrieved clause is a generation problem. An obsolete clause still marked current is a governance problem.
Measure each stage separately, then inspect the end-to-end task. These measures serve different purposes.
| Stage | Useful measure | What to inspect when it fails |
|---|---|---|
| Retrieval | Recall@K against labeled relevant passages; retrieval latency | Missing source, parsing errors, retrieval strategy, filters |
| Context assembly | Evidence coverage; duplicate and stale passages | Lost exceptions, excess expansion, version selection |
| Generation | Faithfulness to evidence; relevance; citation support | Unsupported claims, incorrect synthesis, misleading citations |
| End-to-end | Task completion, refusal quality, user feedback, tail latency | Workflow outcome, escalation, multi-turn consistency |
Recall@K is meaningful only relative to the relevance labels and retrieval unit you define. Faithfulness also has a narrower meaning than factual correctness. An answer can be faithful to a source that is wrong.

RAGAs provides a research basis for evaluating context relevance, faithful use of evidence, and generated-answer quality separately. Automated LLM scoring can help run larger regression sets, but judges can make mistakes. Research on LLM-as-a-judge documents position and verbosity biases as well as reasoning limitations. Calibrate automated scores against human-reviewed examples, preserve scoring prompts and model settings, and send disputed or high-risk cases to reviewers.
Run the same regression set after changing a parser, index, retrieval strategy, prompt, or model. Add production failures to a reviewed backlog without silently changing the benchmark. Use a cadence appropriate to document churn and business risk, and retain source and configuration snapshots so a regression can be reproduced.
Give knowledge an owner and a lifecycle
An ingestion pipeline needs more than successful uploads. Treat unstructured data preparation as parse, transform, and index, with source identity and permissions carried through every transformation. This is a useful engineering shorthand, not a claim that PTI is a universal industry standard.
Parsing should preserve headings, tables, page anchors, and the relationship between an exception and its rule. Transformation should normalize formats and flag duplicates while retaining provenance. Indexing should preserve version and access metadata so later retrieval can select the appropriate evidence.

Assign an owner to each source collection. Retain effective dates, superseded versions, publication status, and source anchors. When a policy changes, track ingestion and indexing completion before treating the new version as searchable. Invalidate affected caches and derived summaries as part of that change.
Entity normalization and relationship extraction can help surface possible contradictions. They cannot reliably decide every conflict. Two passages may refer to different jurisdictions, customer groups, or time periods. A reviewer or explicit business rule must determine which applies. Preserve both the original statements and the resolution history.
Structured knowledge entries can make recurring rules easier to maintain. Each entry should link to its sources, identify an owner, and record when it was reviewed. LLM-generated summaries and wiki-style consolidation are derived material; they require validation and must not silently replace authoritative documents.

In pharmaceutical information or legal research, a newer source may change what a user should rely on. Treat those as high-risk information workflows with qualified professional review where needed. Document retrieval does not establish clinical or legal validity.
Record evidence decisions before generation
An evidence-checking stage between retrieval and generation can make selection decisions easier to inspect. Retrieval supplies candidates. The checking stage applies source, date, applicability, and access rules. Generation receives the selected passages together with their provenance.
A reranker primarily estimates relevance. An evidence-checking stage can additionally test whether a passage is still applicable, whether the user may access it, and whether another retrieved passage contradicts it. Log the decision and its reason, including evidence that was excluded.

Deterministic checks can validate an expiry field or enforce a defined access policy. They cannot establish that an underlying claim is true. LLM-based contradiction detection and source interpretation remain probabilistic and can fail. An evidence decision layer does not guarantee deterministic truth, complete conflict detection, or hallucination-free answers.
For the policy example, prefer the currently effective clause only after checking that its scope matches the request. If two apparently authoritative clauses disagree, retain the conflict and escalate or answer with an explicit limitation. If supporting evidence is missing, ask for clarification or decline to assert the answer.
This separation can reduce irrelevant context and support error attribution. Measure its actual benefit and overhead. An extra validation call can increase latency and cost, and an overly aggressive filter can remove necessary evidence.
Retrieve small passages and read enough context
Evaluation often exposes a granularity mismatch. A small passage can match a query well while omitting the condition that makes its answer valid. Use the match to locate evidence, then expand to the parent section or related passages when needed. Reapply authorization and relevance checks to the expanded material.
Parent-child retrieval is one implementation. Hierarchical summary retrieval is another: RAPTOR recursively embeds, clusters, and summarizes passages into a tree. This differs from simply following a document's table of contents. Document-structure navigation, including approaches such as PageIndex, depends on the quality of the document structure and the navigation process. These are options to evaluate, not interchangeable product features.

Longer context windows can hold a complete section or evidence collected across several retrieval steps. They do not ensure that the model will use every passage correctly. Lost in the Middle found position-sensitive performance in the models and tasks studied. Test evidence placement, coverage, and ordering with your chosen model rather than treating that finding as a universal limit for every current model.
Graph-based retrieval is useful to test when answers require relationships across documents. Microsoft's GraphRAG documentation describes entity and relationship extraction, community construction, and community summaries, with local and global query modes. GraphRAG is a family of approaches; Personalized PageRank is not a required algorithm for every implementation.
Extracted relations and summaries can introduce errors. Preserve links to original passages and review the evidence supporting a synthesized answer. The GraphRAG repository warns that indexing can be expensive. There is no universal token-cost multiplier: measure extraction, summarization, refresh, and query costs on your own corpus. Combining hierarchical and graph retrieval is an architectural option, not an established best choice for every workload.
Budget for accepted answers and knowledge changes
Quality, latency, and cost form a workload-specific trade-off. Establish a simple retrieval baseline before adding multi-hop retrieval, graph expansion, or larger context assembly. Compare alternatives on the same evaluation set and user-role constraints.
Route direct lookups through a lighter path when they meet the acceptance criteria. Reserve iterative retrieval or graph-based synthesis for questions that need it. Precompute stable summaries offline, with explicit refresh and invalidation rules. Bound retrieval rounds, context size, generation length, and elapsed time. Define what happens when the budget is exhausted, including a partial answer, clarification, or escalation.

Track ingestion and reindexing cost alongside online retrieval, reranking, checking, and generation. Include human review and cache invalidation in operational planning. Report cost per accepted task with the acceptance rule stated, plus tail latency and failure rates by question type. A low token bill is not useful if the system fails more tasks or sends unnecessary work to reviewers.
Caching requires the same governance as retrieval. A cache key and invalidation policy should account for authorization scope and relevant knowledge changes. Sharing a cached answer across users without validating permissions can bypass otherwise sound retrieval controls.
Test access control beyond the retriever
Enforce authorization before protected content reaches the model. Carry access controls through parent-context expansion, graph traversal, summaries, logs, and caches. Test denied access explicitly, including permission revocation and sources with mixed visibility.
Retrieved documents are untrusted input. They may contain prompt injection, poisoned content, or embedded instructions that try to change the assistant's behavior. Apply data minimization, source validation, and tool authorization, and keep retrieved instructions separate from system instructions. The OWASP GenAI Security Project provides security guidance for these risks.
Audit records should connect a request to its authorized evidence, selection decisions, and answer while protecting sensitive data in the records themselves. Neither retrieval-time filtering nor an evidence-checking stage guarantees complete prevention of unauthorized disclosure. Security testing, operational review, and incident handling remain necessary.
Start with a reviewed failure set
Collect cases with missing support, stale citations, contradictory passages, and incorrect follow-up answers. Label the failing stage, assign an owner, and rerun those cases after each relevant change. Keep the expected outcome explicit, including when the correct result is refusal or escalation.
Use the ADP console to work with your application and confirm the evaluation and knowledge-management options available to your account. The architecture patterns in this article may require additional components and integration. For the underlying parsing, chunking, and retrieval pipeline, see the Enterprise RAG Guide.

Start building today
If you need more support, please contact us


