Determinism and Reproducibility
Determinism means identical inputs and execution conditions produce identical outputs. Reproducibility means a run can be reconstructed closely enough to debug drift. In generative systems, this spans sampling and decoding, context construction, retrieval, tools, serving, and validators. Most production systems should optimize for reproducibility even when perfect determinism is impossible.
What to record for a replayable run
A run record should include model identifier, prompt messages, decoding parameters, seed if exposed, retrieved chunk IDs, tool schemas, tool outputs, validator versions, and post-processing code. Temperature zero narrows sampling but does not freeze hosted infrastructure or retrieval state; see temperature and determinism.
Reproducibility fails whenever an unrecorded dependency changes. In a RAG system, the same prompt can produce a different answer after chunking, embedding, reranking, or source documents change. In an agent, a tool response, clock, permission state, or retry path can change the final answer even when the model settings are fixed.
LangChain traces help identify model, retriever, tool, and middleware changes. LangGraph checkpoints go further by preserving graph state at execution boundaries, which is useful for replaying long-running or human-reviewed workflows.
| Layer | What to record | Why it matters |
|---|---|---|
| Model | provider, model id, version or deployment name | hosted models can change behind stable names |
| Decoding | temperature, top-p, seed, max tokens | controls stochastic output choices |
| Prompt and context | messages, retrieved chunks, ordering, truncation | defines the actual input distribution |
| Tools | schema versions, arguments, outputs, errors | tool state can dominate the result |
| Validators | schema, citation, safety, and policy versions | post-processing can accept or reject outputs |
A minimal run record
{
"model": "provider-model-version",
"decoding": { "temperature": 0, "top_p": 1 },
"retrieved_chunk_ids": ["policy-7", "policy-9"],
"tool_schema_hash": "sha256:...",
"validator": "citation_support_v3"
}This is the minimum trace needed for agent evaluation.
Replay versus live reproducibility
| Mode | Purpose | Example |
|---|---|---|
| Trace replay | debug a past run with frozen inputs and observations. | rerun the same prompt and retrieved chunk IDs. |
| Live canary | detect drift in the current system. | ask stable benchmark questions every deploy. |
| Deterministic unit test | validate code around the model. | schema validator rejects malformed output. |
| Statistical regression | detect quality drift over a set. | compare pass rate over 200 RAG cases. |
Replay explains what happened. Canaries reveal what would happen now. A mature system needs both.
Realistic drift example
A refund answer changes although temperature is zero. The trace shows the prompt template and model route are unchanged, but retrieved_chunk_ids changed from refunds-007 to refunds-2025-legacy after a re-index. The issue is retrieval state, not decoding randomness. Without retrieval IDs and index versions, the team might incorrectly blame the model.
Caveats
Caching can mask nondeterminism during tests and then disappear in production. Conversely, strict replay can hide live-system drift. Keep both: replay traces for debugging, and live canaries for detecting retrieval, tool, and serving changes.
References
Nav