Why Long Context Does Not Replace RAG
Compare RAG and long-context prompting through parsing, chunking, evidence quality, and governance. Choose an architecture based on your corpus and workload.
A larger context window can hold more documents. It does not decide which version is current, which passages answer a question, or which records a user may access. Those decisions still need an information pipeline.
For a small, stable document set, putting the full text into a prompt may be the simplest approach. As the collection grows or changes, retrieval can select a manageable body of evidence for the model. The useful comparison is between complete systems, including their quality, latency, cost, freshness, and access controls.
Two pipelines connect documents to answers
A conventional retrieval-augmented generation (RAG) system separates document preparation from answering questions. The preparation pipeline parses documents, splits them into chunks, creates embeddings, and indexes the results with source text and metadata. The query pipeline retrieves relevant material and supplies it to the model as evidence.

Query rewriting can help when the user's wording differs from the documents. Hybrid retrieval combines semantic matching with keyword search; a reranker can then reorder candidate passages before context assembly. These are design options, and their benefit should be measured against representative questions. A source citation helps readers inspect an answer, but its presence alone does not establish that the answer is supported.
The existing Enterprise RAG Guide covers the pipeline and retrieval components in more detail. Here, the focus is on the boundaries that a larger context window does not automatically resolve.
Parsing determines what evidence survives
Retrieval can only work with the information that reaches the index. A financial table loses meaning if its values become detached from column headings. A multi-column page can become misleading if extraction interleaves unrelated paragraphs. A text-only pipeline may miss the essential relationship shown in a diagram.
The difficult inputs include merged cells and tables spanning pages, diagrams and charts, formulas, and scans affected by blur or skew. These are document-understanding problems before they become retrieval problems. Increasing the model's context capacity does not restore information discarded upstream.
There are several ways to parse these inputs. Rules, native text extraction, layout analysis, and conventional OCR can be efficient on predictable documents. Specialized document models and multimodal models offer other options for complex pages. Their quality and runtime costs depend on the implementation and the documents; neither route is universally better. The Docling Technical Report describes one modular approach using specialized layout and table-structure models.

A possible architecture is to route straightforward pages through a lighter pipeline and escalate difficult pages to a model-based parser. Evaluate the resulting reading order, table structure, and retained source locations. This is an engineering recommendation, not a claim that ADP exposes this exact routing mechanism.
Chunk size balances focus and completeness
Small chunks can isolate a useful fact for retrieval, but omit its definition, exception, or surrounding explanation. Larger chunks preserve more context, while potentially bringing unrelated material into the result. Neither effect is guaranteed: document structure, embedding behavior, and the questions all influence the outcome.

Parent-child retrieval is one way to separate these needs. Search over smaller passages, then supply the containing section or document when assembling the model's context. Semantic boundaries and limited overlap are other options. There is no universal token count that makes a chunk correct for every corpus.
Chunk fragmentation and Lost in the Middle describe different failure modes. Fragmentation removes or separates needed context during preparation or retrieval. In Liu et al.'s study, relevant information was already present in the input, but the tested models often performed worse when it appeared in the middle rather than near the beginning or end. The study concerns position-sensitive use of context, not a synonym for bad chunking or proof that every newer model behaves identically.
For evaluation, check both whether the necessary evidence was retrieved and whether the model uses it correctly at different positions in the assembled input.
Five limits of a one-pass RAG design
A simple retrieve-once, generate-once pipeline is a useful baseline. Its limitations help explain why teams add more selective retrieval and richer context assembly.
Single-hop retrieval can miss dependencies. A question that combines a lease condition with information from several tenant records may require multiple lookups. Decomposing the question or retrieving iteratively is worth testing when one pass fails to gather all the evidence.
Isolated chunks can lose their meaning. A retrieved clause may depend on a definition elsewhere in the document. Expanding to a parent section can help, provided the additional context remains relevant.
Semantic similarity cannot reliably replace structured computation. Questions such as “Which location has the highest score?” or “How many records meet these conditions?” need exact filtering or aggregation. A structured query or calculation tool is often more appropriate than vector search alone. Its results can still become evidence for the generated answer.
Similarity does not establish validity. An obsolete policy and its replacement may be nearly identical. If both remain searchable, semantic ranking alone can return the superseded version. Preserve version and effective-date metadata, and distinguish a question about current policy from a historical question. New documents also need to complete ingestion before retrieval can find them.
A fixed retrieval step can be unnecessary or insufficient. Some questions can be answered from the evidence already supplied; others need another search. Adaptive or agentic retrieval can vary the process, but adds routing decisions, latency, and failure modes to evaluate.
These are general architectural considerations. They should not be read as a list of automatically enabled ADP capabilities.
Choose retrieval, long context, or both
Retrieval narrows the evidence set. A long context window allows the model to consider more of that evidence together. Combining the two can help when a query spans multiple sections or documents, but direct long-context prompting remains a valid alternative.

Research supports a conditional choice. In Li et al.'s RAG and long-context comparison, sufficiently resourced long-context configurations outperformed RAG on average across the tested datasets, while RAG had a cost advantage. The authors also evaluated routing queries between the approaches. Those results describe their models and benchmark settings, not a permanent ranking for every application.
For a bounded, stable collection that fits comfortably into the available window, direct prompting can avoid maintaining a retrieval index. It is also a useful baseline when the task needs broad coverage of the supplied documents. Measure actual runtime cost and latency, including any caching behavior, rather than assuming that a longer prompt is always uneconomic.
For larger or frequently changing collections, retrieval can limit each request to relevant material. Freshness still requires a working update process and explicit handling of superseded content. If users have different permissions, enforce authorization before evidence enters the prompt, whether that evidence comes from search or full-document loading. A context window is not an access-control mechanism.
A combined design can retrieve authorized, current documents and then supply larger sections for cross-document reasoning. Compare it with direct prompting on the same questions. Include incomplete evidence, conflicting versions, permission boundaries, and questions that should remain unanswered. Review citation support as well as answer quality and runtime behavior.
Long context can replace retrieval for some workloads. It does not remove the need to select, prepare, and govern evidence across all enterprise workloads. Start with the least complex design that meets those requirements, then add retrieval or context expansion where testing shows a need.
Test with your own documents
Use a business manual or policy document and a set of real questions to inspect the evidence behind each answer. Include a question that spans sections and one affected by a document revision. To begin building a knowledge assistant, open the ADP console.

Start building today
If you need more support, please contact us


