Context Construction

Context construction is the packing layer between retrieval and generation. It decides which instructions, tool schemas, conversation turns, chunking outputs, and formatting constraints reach the model. It is one of the most important quality levers in a RAG or agent system because the model can only ground on evidence that is actually visible in the final request.

Packing under a token budget

With a token budget $B$, each candidate item has cost $c_i$ and estimated utility $u_i$. The system chooses a subset with $\sum c_i\le B$, while reserving room for the answer and preserving instruction precedence. This is why retrieval pipelines should return ranked, source-labeled chunks rather than raw documents.

In practice, context construction is a constrained packing problem with hard requirements. System policy, safety instructions, output schema, and required tool definitions are not optional utility items; they are reserved budget. Retrieved evidence and conversation history compete for what remains.

Worked packing table

With a 420-token budget, a simple utility-per-token packer ranks items as follows:

Item	Tokens	Utility	Utility per token	Kept?
system	80	10	0.125	yes
schema	120	9	0.075	yes
retrieved_A	140	8	0.057	yes
retrieved_B	110	6	0.055	no
chat_history	160	5	0.031	no

The kept items use $80+120+140=340$ tokens, leaving 80 unused because the next candidate would exceed the 420-token budget. The greedy packer kept high-utility instructions and evidence but dropped chat history. That trade-off should be visible in determinism and reproducibility traces.

Trust and precedence

Context is not a flat bag of text; it has an authority order. System and developer instructions outrank retrieved documents, which outrank user-supplied text, which outrank tool output. The packer must preserve that order and label each block’s trust level, so a retrieved passage or a user message can never silently override an instruction. That ordering is the core defense against prompt injection, and it must survive truncation: when the budget forces cuts, drop low-utility evidence, never the instruction hierarchy.

Context blocks

Block	Keep when	Drop or compress when
System policy	always	almost never; shorten only by versioned template changes.
Developer task instructions	always for the route	route changes or task changes.
Output schema	downstream software depends on it	use a smaller schema or separate extraction route.
Tool schemas	tool may be called in this state	tool not authorized or irrelevant.
Retrieved evidence	supports the current question	low score, stale, duplicate, or unsupported by permissions.
Conversation history	needed for reference resolution	summarize or drop unrelated turns.
User-provided documents	needed as data	untrusted or too large; summarize with provenance.

Realistic packing failure

A user asks: “Does an enterprise refund of 650 EUR need manager approval?” Earlier in the conversation, the model and user discussed an older policy where the threshold was 500 EUR. The packer then fills most of the available budget with 20 chat turns plus two stale policy chunks, and the current July 2026 policy table is the item that gets truncated.

Step by step, that happens as follows:

1. The packer starts with the current question and the mandatory instructions, schemas, and safety blocks.
2. It then adds a long stretch of conversation history because those turns appear relevant to the refund topic.
3. Two older policy chunks also score well enough to be kept, because they mention approval thresholds and look similar to the current policy.
4. The budget is almost exhausted before the current July 2026 policy table is considered.
5. When the packer reaches that current table, there is not enough budget left, so it gets dropped even though it is the decisive source.

The model then answers, “Yes, manager approval is required above 500 EUR.” That answer is wrong for the current policy, but it is understandable from the packed context: the stale history still contains the old threshold, while the decisive current table never reached the model. The failure is not that the model ignored the right evidence; the failure is that the right evidence was not included in the final context.

A robust trace should show which chunks were considered, which were packed, which were dropped, and why. Without that trace, teams often blame generation when retrieval or packing caused the unsupported answer.

Design rules

• Reserve budget for instructions, schemas, and the expected answer before packing evidence.
• Keep source IDs and metadata next to every chunk.
• Deduplicate near-identical chunks so repeated boilerplate does not crowd out decisive evidence.
• Prefer current, authoritative, and permissioned sources over semantically similar stale sources.
• Label untrusted user or web content as data.
• Log dropped high-scoring evidence for debugging.

Caveats

More context can hurt when it includes conflicting passages or untrusted user text. Label trusted documentation separately from user-provided content. Long context also increases latency and can dilute attention, so the best context is usually selective rather than maximal.

References

• OpenAI API documentation: Text generation
• Lewis et al., 2020, Retrieval-Augmented Generation

Nav

Section — Generative AI and Agentic Systems

← Reranking Grounding →