Continuous Evaluation and Golden Dataset Curation in Production LLM Systems: Architecture, Log Mining, Synthetic Perturbation, and Semantic Drift Quality Gates

In production machine learning systems, offline benchmarks suffer from rapid entropy. While engineering teams frequently launch LLM applications backed by static test suites (curated CSVs of 50 to 100 sample prompts, academic benchmarks like MMLU, or synthetic question-answer pairs), these static datasets fail to predict real-world production performance within weeks of deployment.

As real-world users interact with systems, query distributions shift, prompt engineering patterns evolve, and underlying upstream model APIs change behavior silently. When evaluation sets remain frozen, teams experience the classic evaluation failure mode: high offline benchmark scores paired with climbing production defect rates.

Building a dependable LLM system requires moving from episodic, static testing to a closed-loop continuous evaluation architecture. This involves continuously capturing production telemetry, mining negative user signals, semantically deduplicating edge cases, synthesizing adversarial perturbations, and gating CI/CD pipelines against statistically calibrated golden datasets.

1. The Dataset Drift vs. Judge Drift Problem

Maintaining reliable continuous evaluation in production LLM systems requires resolving two distinct failure modes:

1. Dataset Drift: The semantic distribution of live production traffic diverges from the curated test set. New query formulations, novel user failure modes, seasonal vocabulary changes, and new edge cases emerge in production but are absent from the evaluation suite.
2. Judge Drift: The evaluation mechanism itself changes over time. When using LLM-as-a-judge frameworks (Zheng et al., 2023), upstream provider updates or stochastic sampling variations alter the judge's scoring distribution. Recent research into anytime-valid attribution in LLM pipelines (arXiv:2606.15474) emphasizes that without fixed anchor references, teams cannot determine whether a score drop indicates an application regression or a drift in the evaluator.

A production continuous evaluation control plane must decouple these variables by stabilizing judge calibrations while dynamically refreshing the golden dataset through structured feedback loops.

2. End-to-End Control Plane Architecture

A production-grade continuous evaluation control plane operates across five coordinated layers:

+-----------------------------------------------------------------------------+
|               Continuous Evaluation & Golden Dataset Architecture           |
+-----------------------------------------------------------------------------+
|                                                                             |
|  [Production Telemetry Ingestion]                                            |
|  * OpenTelemetry GenAI Spans (Prompt, Context, Tool Calls, Generation)      |
|  * User Interactions (Feedback, Session Rewrites, Tool Errors, Latency)     |
|         |                                                                   |
|         v                                                                   |
|  [Negative Signal & Anomaly Miner]                                          |
|  * Explicit: Thumbs-down, user edits, reported hallucinations               |
|  * Implicit: Immediate reformulations (<30s), tool retries, schema aborts   |
|         |                                                                   |
|         v                                                                   |
|  [Semantic Clustering & Stratification Engine]                              |
|  * Dense Embedding Projections (BGE / OpenAI Text-Embedding-3)              |
|  * HDBSCAN / K-Means Clustering to prevent over-representation              |
|  * Maximal Marginal Relevance (MMR) Diversity Sampling                      |
|         |                                                                   |
|         v                                                                   |
|  [Synthetic Perturbation & Adversarial Generator]                           |
|  * Constraint Negation, Noise Injection, Multi-Turn Reordering              |
|  * Distractor Context Injection (Needle-in-a-Haystack Stressing)             |
|         |                                                                   |
|         v                                                                   |
|  [Human-in-the-Loop & Judge Calibration]                                    |
|  * Active Learning Triage Queue for Human Review                            |
|  * Inter-Annotator Reliability (Cohen's Kappa / Fleiss' Kappa) Calibration  |
|         |                                                                   |
|         v                                                                   |
|  [Golden Dataset Registry & CI/CD Gating]                                   |
|  * Versioned Test Suites (Tier-1 Smoke vs. Tier-2 Stratified Full Suite)    |
|  * Sequential Probability Ratio Tests (SPRT) on Pull Requests               |
+-----------------------------------------------------------------------------+

3. Production Telemetry and Negative Signal Mining

Continuous evaluation begins at the telemetry layer. Systems must capture structured execution spans following standards like OpenTelemetry GenAI Semantic Conventions.

A standard evaluation trace payload must capture:

• Invocation Metadata: Timestamp, model checkpoint identifier, temperature, top-p, prompt template version.
• Context Artifacts: Retrieved document chunks, similarity scores, reranker scores, system instructions.
• Agent Execution Graph: Tool invocation parameters, tool return codes, execution latency, retry counts.
• Raw Outputs & Token Usage: Prefill token count, completion token count, output text, stop reason.

Negative Signal Detection

Evaluating 100% of production traffic with frontier LLM judges is economically impractical and adds unnecessary noise. Instead, mining pipelines apply heuristic and behavioral filters to isolate high-value failure candidates:

from dataclasses import dataclass
from typing import Optional, List, Dict, Any

@dataclass
class ProductionTrace:
    trace_id: str
    user_input: str
    retrieved_context: List[str]
    model_output: str
    tool_calls: List[Dict[str, Any]]
    explicit_thumbs_down: bool
    session_duration_sec: float
    immediate_user_reformulation: bool
    schema_validation_failed: bool
    tool_error_count: int

def score_trace_for_curation(trace: ProductionTrace) -> float:
    """
    Computes priority score for adding a production trace to the eval triage queue.
    Returns float between 0.0 and 1.0.
    """
    priority = 0.0
    
    # Explicit user signals have highest weight
    if trace.explicit_thumbs_down:
        priority += 0.50
        
    # Structural failures (schema errors, tool exceptions)
    if trace.schema_validation_failed:
        priority += 0.30
    if trace.tool_error_count > 0:
        priority += min(0.25, trace.tool_error_count * 0.10)
        
    # Behavioral heuristics: user rewrote the query immediately after output
    if trace.immediate_user_reformulation:
        priority += 0.20
        
    return min(1.0, priority)

Traces exceeding a priority threshold (typically >= 0.40) are routed into the curation and deduplication pipeline.

4. Semantic Deduplication and Stratification

A recurring failure mode in production dataset management is frequency bias: high-volume, simple user queries flood the dataset, while rare, mission-critical failure modes remain underrepresented.

To build a representative test suite, teams apply dense embedding clustering and diversity sampling (Langfuse Golden Dataset Guide):

1. Embedding Generation: Candidate inputs and context pairs are mapped into a high-dimensional vector space using models such as text-embedding-3-large or bge-large-en-v1.5.
2. Density-Based Clustering: HDBSCAN or hierarchical k-means groups similar query patterns into distinct behavioral clusters (e.g., entity extraction, multi-hop reasoning, format conversion, out-of-domain rejection).
3. Maximal Marginal Relevance (MMR) Sampling: Within each cluster, items are selected to balance high relevance (closeness to the failure centroid) with high diversity (distance from already selected test cases):

$$\text{MMR}(q, D, S) = \operatorname{argmax}_{d_i \in D \setminus S} \left[ \lambda \cdot \text{Sim}_1(d_i, q) - (1 - \lambda) \max_{d_j \in S} \text{Sim}_2(d_i, d_j) \right]$$

Where:

• $q$ is the cluster centroid vector.
• $D$ is the set of candidate failure traces in the cluster.
• $S$ is the set of already selected golden dataset items.
• $\lambda$ is the diversity tuning parameter (typically set to $0.65$).

1. Tiered Partitioning & Pruning: Golden datasets are organized into stratified partitions with maximum capacity limits (e.g., 20 items per cluster). As new edge cases enter a saturated cluster, an age-decayed FIFO or utility-weighted policy evicts redundant historical cases.

5. Synthetic Perturbation and Boundary Stress Testing

Real-world failures often expose fragile prompt boundaries. To prevent models from merely memorizing specific phrasing, curation pipelines apply synthetic perturbation (Parrish et al., 2021; Wang et al., 2024).

For every validated production failure, an automated worker generates 3 to 5 synthetic variants across distinct perturbation axes:

• Syntactic & Typographical Noise: Introducing realistic keystroke slips, missing punctuation, and grammatical contractions.
• Context Distractor Injection: Appending 3 to 10 irrelevant enterprise documents to retrieved context windows to test retriever and reader resistance to noise.
• Constraint Inversion & Ordering Permutation: Swapping the order of instructions (e.g., placing negative constraints before positive ones) to verify adherence under different attention patterns.
• Adversarial Jailbreak Probing: Generating semantic paraphrases that attempt to bypass safety and role boundaries.

{
  "golden_case_id": "eval-rag-sec-8841",
  "base_query": "What is the SOC2 compliance policy for AWS S3 bucket encryption?",
  "synthetic_variants": [
    {
      "variant_type": "context_distractor_stress",
      "modified_query": "What is the SOC2 compliance policy for AWS S3 bucket encryption?",
      "injected_distractors": ["azure_blob_storage_encryption_v2.pdf", "marketing_brand_guidelines_2026.docx"]
    },
    {
      "variant_type": "constraint_permutation",
      "modified_query": "Do not mention Azure. Explain the SOC2 policy for AWS S3 bucket encryption in under 50 words using bullet points."
    }
  ]
}

6. Judge Calibration and Quality Control

When evaluation pipelines use LLM-as-a-judge models, the evaluation engine must prevent judge drift (arXiv:2606.15474).

Preventing Judge Drift

1. Pinned Model Versions: Never use rolling aliases (such as gpt-4o or claude-3-7-sonnet) for automated scoring in CI/CD. Pin immutable snapshot identifiers (e.g., gpt-4o-2024-08-06 or exact inference engine container digests).
2. Deterministic Sampling: Set temperature=0.0 and enforce structured output schemas for judge outputs.
3. Anchor Calibrations: Maintain a fixed, immutable baseline set of 50 human-graded responses. Before any CI run, evaluate the judge against the anchor set. If Cohen's kappa coefficient ($\kappa$) between the automated judge and the human ground truth drops below $0.80$, the test suite halts with a JudgeCalibrationError.

$$\kappa = \frac{P_o - P_e}{1 - P_e}$$

Where $P_o$ is the observed relative agreement between the judge and human labelers, and $P_e$ is the hypothetical probability of chance agreement.

7. CI/CD Integration and Regression Gating

Continuous evaluation culminates in automated build gates. Every pull request that modifies system prompts, retrieval parameters, agent tools, or foundation model versions triggers an evaluation workflow against the versioned golden dataset (Daftuar, 2026).

Two-Tiered Evaluation Strategy

To balance pull request turnaround times with statistical coverage, production systems utilize a two-tier evaluation matrix:

• Tier-1 Smoke Suite: 50 curated critical edge cases executed on every pull request commit. Latency target is under 2 minutes. Focuses on exact schema matches, tool parameter validity, deterministic assertions, and zero-shot safety constraints.
• Tier-2 Stratified Suite: 500 to 2,000 cluster-sampled items executed on nightly builds or pre-release release branches. Latency target is 15 to 45 minutes. Evaluates RAGAS faithfulness, context recall, semantic similarity, toxicity, and multi-step tool trajectory efficiency.

Statistical Regression Detection

Naive point comparisons (e.g., failing a build if average faithfulness drops from 0.94 to 0.93) result in high false-positive rates due to model stochasticity.

Production gates apply Sequential Probability Ratio Testing (SPRT) or bootstrap confidence intervals (typically 1,000 resamples at $\alpha = 0.05$). A regression is flagged only when the lower bound of the difference distribution ($\Delta = \mu_{\text{candidate}} - \mu_{\text{baseline}}$) falls below the acceptable margin of error:

# Example CI/CD Evaluation Execution Step
python -m eval_runner \
  --dataset s3://llm-eval-registry/golden/production_v4.2.jsonl \
  --candidate-prompt src/prompts/agent_system_v3.jinja2 \
  --baseline-version prod-current \
  --alpha 0.05 \
  --max-tolerated-drop 0.02 \
  --output eval_report.json

8. Summary Checklist for Production Eval Pipelines

Teams deploying continuous evaluation control planes should verify the following operational guardrails:

• [ ] Telemetry logs capture raw prompts, full retrieved context strings, tool invocations, and user interactions via OpenTelemetry semantic conventions.
• [ ] Negative signal mining automatically extracts user reformulations, schema errors, and feedback downvotes.
• [ ] Golden datasets are clustered via dense embeddings and sampled with MMR to prevent frequency bias.
• [ ] Synthetic perturbations stress test models against distractors, typos, and reordered instructions.
• [ ] LLM judges run on pinned model versions with deterministic decoding and regular anchor calibrations.
• [ ] CI/CD gates evaluate PRs using statistical hypothesis testing rather than uncalibrated point-estimate comparisons.

Sources

• Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines (arXiv:2606.15474)
• Evaluating Agentic AI in the Wild: Failure Modes, Drift Patterns, and a Production Evaluation Framework (arXiv:2605.01604)
• A Survey of Useful LLM Evaluation (arXiv:2406.00936)
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv:2306.05685)
• Ragas: Automated Evaluation of Retrieval Augmented Generation (arXiv:2309.15217)
• BBQ: A Hand-Built Bias Benchmark for Question Answering (arXiv:2110.08193)
• Measuring Massive Multitask Language Understanding - MMLU (arXiv:2009.03300)
• OpenTelemetry Semantic Conventions for Generative AI Systems
• Langfuse: Golden Dataset Evaluation Engineering Guide
• LLM Evaluation in Production: Building the Eval Pipeline That Runs on Every Deploy (Alok Ranjan Daftuar)