Online Evaluation and Multi-Armed Bandit Routing in Production LLM Systems: Interleaving, Counterfactual Estimation, and Adaptive Traffic Allocation

Static offline benchmarks such as MMLU, HumanEval, and synthetic LLM-as-a-judge evaluation pipelines have become standard fixtures in modern AI development. However, production engineering teams frequently observe that offline benchmark improvements fail to translate into tangible user satisfaction or business outcomes. Static evaluation suites suffer from test-set contamination, lack multi-turn conversational realism, and cannot capture subjective user preferences, interaction latencies, or workflow fatigue.

To deploy, compare, and optimize models safely in production, systems require continuous online evaluation. Yet traditional randomized A/B testing introduces significant drawbacks: static 50/50 traffic splits impose substantial cumulative regret, exposing large user cohorts to inferior or unnecessarily expensive model variants for weeks while waiting for statistical significance.

Modern production architectures address these challenges by pairing continuous online evaluation methodologies—such as Team Draft Interleaving and implicit telemetry mining—with contextual multi-armed bandit routing and counterfactual off-policy evaluation. This article examines the architectural foundations, mathematical formulations, and engineering implementations required to build an adaptive, low-latency online evaluation and routing system for large language models.

The Failure Modes of Offline Evaluation and Static A/B Testing

Evaluating generative language models prior to deployment typically relies on three paradigms:

1. Curated Ground-Truth Benchmarks: Deterministic accuracy metrics on fixed test sets (e.g., exact match, BLEU, pass@k on coding tasks).
2. Synthetic LLM Judges: Using frontier models to score prompt-response pairs along rubric criteria such as helpfulness, conciseness, and tone.
3. Pre-Deployment Human Annotation: Manual side-by-side labeling of sampled model completions.

While valuable during pre-training and supervised fine-tuning, these approaches exhibit structural blind spots once exposed to live production workloads.

+-------------------------------------------------------------------------------+
|                       The Production Evaluation Gap                           |
+-------------------------------------------------------------------------------+
|  Offline Benchmarks               Real Production Telemetry                   |
|  ------------------               -------------------------                   |
|  * Static prompt distributions    * Dynamic, shifting multi-turn context      |
|  * No latency/UX sensitivity      * Dwell time, edit distance, cancellations  |
|  * High risk of data leakage      * Real economic constraints ($/1M tokens)   |
|  * Artificial scoring rubrics     * Direct user task completion and churn     |
+-------------------------------------------------------------------------------+

Static Distribution Shift and Feedback Latency

Offline test sets represent a static snapshot of past traffic. In customer-facing applications, user query distributions drift rapidly in response to product updates, external events, and evolving user behavior. Furthermore, synthetic evaluators often exhibit systematic biases, including verbosity bias, self-enhancement bias, and sensitivity to prompt phrasing.

The Opportunity Cost of Naive A/B Testing

When migrating from offline evaluation to production testing, teams historically default to randomized A/B experiments. In an A/B test with $K$ candidate models or prompt chains:

• Traffic is partitioned into static buckets (e.g., 50% Control, 50% Variant).
• Each variant receives fixed traffic until the sample size reaches statistical power for a target minimum detectable effect (MDE).

In high-throughput LLM applications, this approach incurs two severe costs:

1. Regret Overhead: If Variant B is inferior (higher hallucination rate or degraded response formatting), 50% of active users experience that degradation across the entire duration of the test.
2. Economic Waste: When testing a high-cost frontier model against a lightweight fine-tuned model, static allocation sends millions of simple queries to the expensive model, burning compute budget without delivering proportional user value.

Interleaving for Generative Model Evaluation

In information retrieval and search ranking, interleaving has long served as a high-sensitivity alternative to traditional A/B testing. Rather than splitting users into separate buckets, interleaving blends candidate rankings into a single interface presentation, allowing within-subject comparisons that drastically reduce variance.

For generative LLM applications—such as multi-candidate code completions, conversational suggestion chips, and multi-result RAG search—interleaving techniques can be adapted to evaluate candidate models with significantly higher statistical power.

Team Draft Interleaving (TDI)

Originally formulated by Chapelle et al. (2012), Team Draft Interleaving simulates a team sports draft to construct an unbiased combined list from two ranking engines $A$ and $B$.

In a generative system presenting $k$ candidates (e.g., alternative code refactorings or query expansions):

1. An empty output list $L$ is initialized.
2. At each selection step $i \in [1, k]$, a coin toss determines which model selects first.
3. Model $A$ adds its top unselected candidate to $L$ and assigns that item to Team $A$.
4. Model $B$ adds its top unselected candidate to $L$ and assigns that item to Team $B$.
5. The process repeats until $L$ contains $k$ items.

       Candidate Model A                    Candidate Model B
     +--------------------+               +--------------------+
     | Suggestion A1 (1)  |               | Suggestion B1 (1)  |
     | Suggestion A2 (2)  |               | Suggestion B2 (2)  |
     | Suggestion A3 (3)  |               | Suggestion B3 (3)  |
     +--------------------+               +--------------------+
               \                                    /
                \     Team Draft Selection Round   /
                 +--------------------------------+
                                 |
                                 v
                     Interleaved Presentation (L)
                 +--------------------------------+
                 | Pos 1: Suggestion A1  (Team A) |  <-- Coin Toss: A first
                 | Pos 2: Suggestion B1  (Team B) |
                 | Pos 3: Suggestion B2  (Team B) |  <-- Coin Toss: B first
                 | Pos 4: Suggestion A2  (Team A) |
                 +--------------------------------+

When a user interacts with an item (e.g., clicking, accepting a diff, or copying text), credit is attributed to the originating team. Because candidate order is randomized at each position step, presentation bias is eliminated.

Probabilistic Multileaving (PM)

When evaluating more than two candidate configurations ($K > 2$), Schuth et al. (2015) extended pairwise interleaving into Probabilistic Multileaving. PM samples candidate items according to a softmax distribution over candidate rankings, preserving historical interaction reusability for off-policy counterfactual analysis.

Mining Implicit and Explicit Production Telemetry

To evaluate generative models in real time, the online evaluation layer must convert unstructured user interactions into calibrated scalar reward signals $r_t \in [0, 1]$.

+-------------------------------------------------------------------------------+
|                      Telemetry Signal Extraction Pipeline                     |
+-------------------------------------------------------------------------------+
|                                                                               |
|  [ User Request ] ---> [ Gateway / Bandit Router ] ---> [ Candidate LLM ]     |
|                                                                   |           |
|                                                                   v           |
|  [ Implicit Telemetry ] <--- [ Client Interface ] <--- [ Streaming Response ] |
|  * Keystroke edit distance                                                    |
|  * Copy-to-clipboard events                                                   |
|  * Query reformulations (<30s)                                                |
|  * Downstream tool execution                                                  |
|           |                                                                   |
|           v                                                                   |
|  [ Reward Extraction Engine ] ---> Composite Reward: r = f(Q, Cost, TTFT)    |
|                                                                               |
+-------------------------------------------------------------------------------+

Explicit Feedback Signals

Explicit user feedback—such as thumbs up/down, numeric ratings, or report flags—provides direct ground truth. However, explicit feedback in production systems is notoriously sparse, typically generated on less than 1% to 3% of total requests. Relying solely on explicit feedback leads to extreme sample selection bias, as users primarily submit feedback during extreme dissatisfaction or exceptional satisfaction.

Implicit Behavioral Telemetry

Implicit telemetry leverages passive user interaction metrics that are recorded automatically for 100% of user interactions:

1. Inline Code Generation / Copilots:

• Acceptance Rate: Whether the completion was accepted via the completion trigger (e.g., Tab key).
• Normalized Levenshtein Retention: Measuring the character-level edit distance between the suggested text $y$ and the code retained in the editor after a time window $\Delta t = 60s$:

$$\text{Retention}(y, y_{\text{final}}) = 1 - \frac{\text{Levenshtein}(y, y_{\text{final}})}{\max(|y|, |y_{\text{final}}|)}$$

• Downstream Compilation and Linting: Whether code generated by the model passes automated syntax checks without immediate user rollback.

1. Conversational and RAG Interfaces:

• Copy / Export Actions: Direct user copy-to-clipboard actions provide a strong positive quality indicator.
• Query Reformulation Rate: If a user submits a follow-up query within 30 seconds containing high lexical overlap with the previous prompt, it indicates that the model's initial answer was insufficient.
• Response Dwell Time vs. Reading Velocity: The ratio of user dwell time on the response view relative to expected reading speed (e.g., 200 words per minute). Extremely short dwell times followed by immediate query termination signal low utility.

Composite Multi-Objective Reward Function

Production routing decisions cannot optimize for generation quality in isolation; they must balance quality against serving latency and inference costs. A production reward function combines these factors into a normalized scalar:

$$r_t = w_q \cdot Q(x, y, \text{telemetry}) - w_c \cdot \left(\frac{\text{Cost}(x, y)}{\text{Cost}_{\max}}\right) - w_l \cdot \left(\frac{\text{TTFT}}{\text{TTFT}_{\max}}\right)$$

Where:

• $Q \in [0, 1]$ represents the composite telemetry quality score.
• $\text{Cost}(x, y)$ is the dollar cost of prompt and completion tokens.
• $\text{TTFT}$ is the Time-to-First-Token in milliseconds.
• $w_q, w_c, w_l$ are weighting hyperparameters satisfying $w_q + w_c + w_l = 1$.

Contextual Multi-Armed Bandits for Adaptive Routing

To minimize cumulative regret while maintaining continuous exploration of new models, production routing architectures employ Contextual Multi-Armed Bandits (MAB).

Unlike context-free bandits that assume a single global reward distribution per arm, contextual bandits observe a feature vector $x_t \in \mathbb{R}^d$ for each incoming prompt and select an action $a_t \in \mathcal{A}$ to maximize expected reward.

       Incoming Prompt Context x_t (Embeddings, Prompt Length, Intent)
                                     |
                                     v
                       +---------------------------+
                       | Contextual Bandit Engine  |
                       | (LinUCB / Thompson Samp.) |
                       +---------------------------+
                        /            |            \
                       /             |             \
                      v              v              v
               [ Arm 1: SLM ]  [ Arm 2: MoE ]  [ Arm 3: Frontier ]
                (e.g., 8B)     (e.g., 8x7B)      (e.g., Claude)
                      \              |              /
                       \             |             /
                        v            v            v
                       +---------------------------+
                       |   Reward Engine r_t(x, a) |
                       +---------------------------+
                                     |
                                     v
                      Online Ridge Parameter Update

Mathematical Formulation: LinUCB

As detailed by Li et al. (2010), the Linear Upper Confidence Bound (LinUCB) algorithm assumes the expected reward for arm $a$ is a linear function of the context vector $x_t$:

$$\mathbb{E}[r_{t, a} | x_t] = x_t^T \theta_a^*$$

Where $\theta_a^*$ is an unknown parameter vector. For each arm $a \in \mathcal{A}$, the system maintains a design matrix $D_a \in \mathbb{R}^{m \times d}$ and observed reward vector $c_a \in \mathbb{R}^m$. Applying ridge regression with regularization parameter $\lambda$:

$$A_a = D_a^T D_a + I_d = \sum_{\tau=1}^t x_\tau x_\tau^T + I_d$$ $$b_a = D_a^T c_a = \sum_{\tau=1}^t r_\tau x_\tau$$ $$\hat{\theta}_a = A_a^{-1} b_a$$

At each decision step $t$, LinUCB selects the arm that maximizes the upper confidence bound:

$$a_t = \arg\max_{a \in \mathcal{A}} \left( \hat{\theta}_a^T x_t + \alpha \sqrt{x_t^T A_a^{-1} x_t} \right)$$

The parameter $\alpha = 1 + \sqrt{\ln(2/\delta) / 2}$ controls the exploration rate:

• $\hat{\theta}_a^T x_t$ represents the estimated mean reward (exploitation).
• $\alpha \sqrt{x_t^T A_a^{-1} x_t}$ represents the standard deviation of the estimate (exploration). As an arm is selected more frequently in a specific context region, $A_a$ grows, shrinking the confidence bound and focusing future traffic on optimal arms.

Contextual Thompson Sampling

In non-stationary environments where model backends are updated or user behavior shifts, Contextual Thompson Sampling (Agrawal & Goyal, 2013) provides a Bayesian alternative. Instead of choosing the deterministic upper confidence bound, parameters are sampled from the posterior distribution:

$$\tilde{\theta}_a \sim \mathcal{N}\left( \hat{\theta}_a, v^2 A_a^{-1} \right)$$ $$a_t = \arg\max_{a \in \mathcal{A}} \left( x_t^T \tilde{\theta}_a \right)$$

Thompson Sampling naturally adapts to multi-modal reward distributions and exhibits smoother exploration characteristics than deterministic UCB policies.

Counterfactual Off-Policy Evaluation (OPE)

When evaluating a newly trained model candidate $\pi_{\text{new}}$ or a revised prompt template, deploying it directly into live production traffic carries risk. Counterfactual Off-Policy Evaluation allows engineering teams to estimate the expected performance of $\pi_{\text{new}}$ using logged interaction data collected under historical routing policy $\pi_0$, without deploying the new model to users.

The Logging Bias Problem

Historical logs contain tuples of $(x_i, a_i, r_i, p_i)$, where $p_i = \pi_0(a_i | x_i)$ is the probability that the logging policy assigned action $a_i$ given context $x_i$. A naive sample average of rewards for cases where $a_i = \pi_{\text{new}}(x_i)$ is heavily biased because the logging policy did not select actions uniformly at random.

Inverse Propensity Scoring (IPS)

To correct for selection bias, Inverse Propensity Scoring (Horvitz & Thompson, 1952) weights observed rewards by the importance ratio:

$$\hat{V}_{\text{IPS}}(\pi_{\text{new}}) = \frac{1}{N} \sum_{i=1}^N \frac{\pi_{\text{new}}(a_i | x_i)}{\pi_0(a_i | x_i)} r_i$$

Under standard unconfoundedness and common support assumptions ($\pi_0(a|x) > 0$ whenever $\pi_{\text{new}}(a|x) > 0$), $\hat{V}_{\text{IPS}}$ is an unbiased estimator of the new policy's expected reward $\mathbb{E}_{x \sim \mathcal{D}, a \sim \pi_{\text{new}}}[r(x, a)]$.

+-------------------------------------------------------------------------------+
|                       Off-Policy Evaluation Trade-Offs                        |
+-------------------------------------------------------------------------------+
|  Estimator              Bias        Variance       Key Weakness               |
|  ------------------     --------    -----------    -------------------------  |
|  Direct Method (DM)     High        Low            Model misspecification     |
|  IPS                    Zero        High / Unb.    Small propensities p_i -> 0|
|  Self-Normalized IPS    Low         Moderate       Slight sample-size bias    |
|  Doubly Robust (DR)     Zero (if 1) Low            Requires reward regression |
+-------------------------------------------------------------------------------+

Doubly Robust Estimation

While unbiased, IPS can suffer from high variance when logging probabilities $\pi_0(a_i | x_i)$ are small. The Doubly Robust (DR) estimator (Dudík et al., 2011) combines a direct reward regression model $\hat{r}(x, a)$ with propensity weighting:

$$\hat{V}_{\text{DR}}(\pi_{\text{new}}) = \frac{1}{N} \sum_{i=1}^N \left( \hat{r}(x_i, \pi_{\text{new}}(x_i)) + \frac{\mathbb{I}(a_i = \pi_{\text{new}}(x_i))}{\pi_0(a_i | x_i)} \left( r_i - \hat{r}(x_i, a_i) \right) \right)$$

The Doubly Robust estimator possesses a fundamental property: it remains statistically unbiased if either the propensity model $\pi_0$ is accurate or the reward regression model $\hat{r}$ is accurate. This property makes DR the standard estimator for validating LLM routing updates in enterprise pipelines.

Production System Architecture and Implementation

Building a scalable online evaluation and bandit routing engine requires coordinating sub-millisecond routing decisions with delayed, asynchronous telemetry ingestion.

                                 [ Production User ]
                                          |
                                   (1) Prompt Request
                                          v
+-------------------------------------------------------------------------------+
|                       API Gateway / Inference Router                          |
|                                                                               |
|  +--------------------+   (2) Feature   +----------------------------------+  |
|  | Context Extraction | --------------> | LinUCB / Bandit In-Memory Model  |  |
|  +--------------------+                 +----------------------------------+  |
|            |                                              |                   |
|            | (3) Route Request                            | (4) Log Action    |
|            v                                              v     & Propensity  |
|  +--------------------+                         +--------------------------+  |
|  | Model Execution    |                         | Logging Event Bus        |  |
|  | (SLM / MoE / API)  |                         | (Redpanda / Kafka)       |  |
|  +--------------------+                         +--------------------------+  |
+--------------------------------------------------------------|----------------+
             |                                                 |
      (5) Response Stream                                      |
             v                                                 |
     [ Client Telemetry ]                                      |
             |                                                 |
      (6) Asynchronous Interaction Log                         |
             \                                                 /
              v                                               v
+-------------------------------------------------------------------------------+
|                      Asynchronous Telemetry Aggregator                        |
|                                                                               |
|  * Reconciles action event with delayed telemetry via Session ID              |
|  * Computes Composite Reward r_t = f(Quality, Latency, Cost)                  |
|  * Publishes training tuples: (x_t, a_t, r_t, p_t)                            |
+-------------------------------------------------------------------------------+
                                         |
                                         v
+-------------------------------------------------------------------------------+
|                       Bandit Model Update Daemon                              |
|                                                                               |
|  * Incremental Ridge Matrix Update: A_a += x x^T, b_a += r x                  |
|  * Atomic parameter broadcast to Gateway in-memory cache every N events       |
|  * Real-time divergence and canary safety monitoring                          |
+-------------------------------------------------------------------------------+

1. In-Memory Contextual Bandit Router

To prevent routing bottlenecks, the bandit policy must execute in under 5 milliseconds. The routing service maintains model matrices $A_a^{-1}$ and vectors $\hat{\theta}_a$ directly in local process memory, eliminating external database network hops during the inference critical path.

import numpy as np
from typing import Dict, Tuple

class ProductionLinUCBRouter:
    def __init__(self, n_features: int, model_arms: list[str], alpha: float = 0.5):
        self.n_features = n_features
        self.model_arms = model_arms
        self.alpha = alpha
        
        # State matrices: A_a = D_a^T D_a + I, b_a = D_a^T r
        self.A: Dict[str, np.ndarray] = {
            arm: np.identity(n_features, dtype=np.float32) for arm in model_arms
        }
        self.A_inv: Dict[str, np.ndarray] = {
            arm: np.identity(n_features, dtype=np.float32) for arm in model_arms
        }
        self.b: Dict[str, np.ndarray] = {
            arm: np.zeros((n_features, 1), dtype=np.float32) for arm in model_arms
        }
        self.theta: Dict[str, np.ndarray] = {
            arm: np.zeros((n_features, 1), dtype=np.float32) for arm in model_arms
        }

    def route(self, context_vector: np.ndarray, min_propensity: float = 0.05) -> Tuple[str, float]:
        """
        Calculates UCB score for each model arm and applies epsilon-floor exploration.
        Returns: (selected_arm, action_probability)
        """
        x = context_vector.reshape(-1, 1)
        p_scores = {}
        
        for arm in self.model_arms:
            theta_a = self.theta[arm]
            A_inv_a = self.A_inv[arm]
            
            # Estimated mean reward + UCB exploration bonus
            mean_reward = float(np.dot(theta_a.T, x))
            uncertainty = self.alpha * float(np.sqrt(np.dot(x.T, np.dot(A_inv_a, x))))
            p_scores[arm] = mean_reward + uncertainty
            
        best_arm = max(p_scores, key=p_scores.get)
        
        # Softmax or epsilon-clamping for logged propensity tracking
        K = len(self.model_arms)
        propensity = (1.0 - min_propensity) + (min_propensity / K)
        
        return best_arm, propensity

    def update(self, arm: str, context_vector: np.ndarray, reward: float):
        """
        Asynchronous Sherman-Morrison rank-1 update to avoid full matrix inversions.
        """
        x = context_vector.reshape(-1, 1)
        self.A[arm] += np.dot(x, x.T)
        self.b[arm] += reward * x
        
        # Sherman-Morrison formula for fast rank-1 inverse update:
        # (A + x x^T)^-1 = A^-1 - (A^-1 x x^T A^-1) / (1 + x^T A^-1 x)
        A_inv_x = np.dot(self.A_inv[arm], x)
        denominator = 1.0 + float(np.dot(x.T, A_inv_x))
        self.A_inv[arm] -= np.dot(A_inv_x, A_inv_x.T) / denominator
        
        # Update point estimate
        self.theta[arm] = np.dot(self.A_inv[arm], self.b[arm])

2. Handling Delayed and Missing Rewards

A primary operational challenge in online LLM evaluation is delayed feedback. A routing decision occurs at $t = 0$, the completion streams at $t = 500\text{ms}$, but user interaction telemetry (edits, copy events, follow-up queries) may arrive minutes later or never arrive.

Production systems resolve this through a two-stage reconciliation window:

• Logging Event: When an arm is selected, the gateway emits an event to a Kafka/Redpanda topic containing session_id, trace_id, context_vector, selected_arm, and propensity_score.
• Telemetry Join Window: An asynchronous worker maintains a sliding join window (e.g., 5 minutes) in Redis or a stream processor (e.g., Apache Flink). If an explicit negative or positive signal arrives, the reward is calculated and emitted to the update queue.
• Default Imputation: If the window expires with no user interaction, a neutral baseline reward is imputed to prevent survival bias (ignoring abandoned sessions).

3. Safety Guardrails and Degradation Failsafes

To prevent the bandit from driving traffic to a failing model backend or unstable prompt version:

• Exploration Flooring ($\epsilon_{\min}$): Never allow action probability to drop to 0. Enforce $\pi(a|x) \ge 0.05$ across all arms to preserve support for counterfactual evaluation.
• Automated Circuit Breakers: If an arm exhibits consecutive API error rates exceeding 5% over a 1-minute rolling window, the gateway temporarily disables the arm and routes traffic to deterministic fallbacks.
• Canary Divergence Bounds: If rolling reward for a candidate arm drops below $3\sigma$ of the baseline control policy, the candidate is automatically evicted from the bandit arm set.

Architectural Comparison: Evaluation Paradigms

| Dimension | Offline Benchmark Suites | Static A/B Testing | Team Draft Interleaving | Contextual Bandit Routing | | :--- | :--- | :--- | :--- | :--- | | Feedback Loop | Pre-deployment, hours/days | Post-deployment, weeks | Real-time, hours | Real-time, continuous | | Traffic Split | 0% production users | Fixed 50/50 or N-way | Unified presentation list | Adaptive, utility-weighted | | Cumulative Regret | Zero (offline) | High (exposes 50% users) | Low (within-subject) | Minimal (sub-linear regret) | | Statistical Power | High (fixed tests) | Low (high variance) | Very High (paired test) | High (context-aware) | | Cost Optimization | None | None | None | Active ($/token constrained) | | Applicability | Pre-training, regression | All application types | Multi-result (Code/RAG) | Point routing, model cascades |

Conclusion

Relying solely on static offline benchmarks leaves production LLM architectures vulnerable to distribution shifts, benchmark overfitting, and undetected user experience degradations. While traditional A/B testing provides empirical validation, its static traffic partitioning incurs severe economic and user-experience regret.

By combining interleaved multi-candidate evaluation, real-time behavioral telemetry mining, and contextual multi-armed bandits, engineering teams can build adaptive routing layers that optimize quality, latency, and cost simultaneously. Paired with counterfactual off-policy evaluation, this infrastructure enables safe, continuous model deployment without risking production stability.

Sources

• Chapelle, O., Joachims, T., Radlinski, F., & Yue, Y. (2012): Large-Scale Validation and Analysis of Interleaved Search Evaluation (ACM TOIS)
• Schuth, A., Oosterhuis, H., Whiteson, S., & de Rijke, M. (2015): Probabilistic Multileave for Online Retrieval Evaluation (ACM SIGIR)
• Li, L., Chu, W., Langford, J., & Schapire, R. E. (2010): A Contextual-Bandit Approach to Personalized News Article Recommendation (arXiv:1003.0146)
• Agrawal, S., & Goyal, N. (2013): Thompson Sampling for Contextual Bandits with Linear Payoffs (arXiv:1209.3352)
• Dudík, M., Langford, J., & Li, L. (2011): Doubly Robust Policy Evaluation and Learning (arXiv:1103.4601)
• Horvitz, D. G., & Thompson, D. J. (1952): A Generalization of Sampling Without Replacement from a Finite Universe (JASA)
• Nguyen, T., et al. (2024): Online Multi-LLM Selection via Contextual Bandits Under Budget Constraints (arXiv:2506.17670)