Model Merging in Production: Comparing Task Arithmetic, TIES-Merging, DARE, and SLERP Architecture, Weight Interference Mitigation, and Serving Economics

In production language model deployment, teams frequently encounter a fundamental operational tension: specialized task performance versus serving cost. Fine-tuning distinct base model instances on specific internal domains (such as code generation, legal document analysis, mathematical reasoning, and safety alignment) produces high task accuracy, but serving multiple independent 70B or 8B parameter models multiplies infrastructure footprint and fractures GPU memory caching.

Traditional solutions to multi-task adaptation present significant trade-offs. Joint multi-task retraining requires assembling massive consolidated datasets and orchestrating multi-node compute clusters, introducing substantial gradient conflict across heterogeneous objectives. Dynamic parameter-efficient fine-tuning (PEFT) architectures like LoRA serving multiplexers (e.g., S-LoRA or Punica) allow adapter swapping on a shared base model, yet introduce non-trivial kernel launch overhead, memory fragmentation, and adapter scheduling complexity under heavy concurrency.

Model merging has emerged as an architectural alternative: combining the weight matrices of multiple fine-tuned models directly in parameter space without requiring retraining, gradient computation, or access to the original training data. Powered by open-source tooling such as MergeKit, engineers can fuse diverse capabilities into a single consolidated checkpoint. However, combining non-linear neural networks in weight space introduces destructive parameter interference, sign conflict, and manifold distortion.

Here is an architectural breakdown of modern model merging techniques, the mathematics of weight interference mitigation, systems-level execution pipelines, and the serving economics of merged models in production.

The Geometry of Parameter Space and Task Vectors

Model merging relies on empirical findings in deep learning loss landscapes: models fine-tuned from a common pre-trained checkpoint remain within the same low-loss basin, connected by linear or low-curvature paths (Wortsman et al., 2022).

Model Soups and Weight Averaging

The earliest manifestation of parameter fusion in fine-tuned models is the "Model Soup" paradigm (Wortsman et al., 2022). If multiple models $\{\theta_1, \theta_2, \dots, \theta_K\}$ are fine-tuned from the identical pre-trained initialization $\theta_0$ using different hyperparameters or data orderings, their weights can be averaged linearly:

$$\theta_{soup} = \frac{1}{K} \sum_{k=1}^K \theta_k$$

While effective for ensembling runs of the same downstream task, uniform weight averaging degrades sharply when attempting to fuse models fine-tuned on divergent, heterogeneous tasks.

Task Arithmetic and Task Vectors

To merge distinct capabilities, Ilharco et al. (2022) formalized the concept of Task Vectors. A task vector $\tau_k$ represents the displacement in parameter space induced by fine-tuning on task $k$:

$$\tau_k = \theta_k - \theta_0$$

Under linear task arithmetic, multiple task vectors are scaled and added back to the pre-trained base checkpoint:

$$\theta_{merged} = \theta_0 + \sum_{k=1}^K \lambda_k \tau_k$$

where $\lambda_k \in \mathbb{R}^+$ represents the task weight scaling coefficient. Task arithmetic also supports task subtraction (e.g., subtracting a toxic behavior vector to debias a model) and task analogy operations.

Despite its mathematical elegance, naive linear task arithmetic encounters severe degradation when scaling beyond two or three tasks, or when merging models fine-tuned across disparate domains.

Naive Task Arithmetic vs. Interference Mitigation
───────────────────────────────────────────────────────────────────
Base Checkpoint (θ₀) ──┐
                      ├──> Task Vector 1 (τ₁ = θ₁ - θ₀) ──┐
                      ├──> Task Vector 2 (τ₂ = θ₂ - θ₀) ──┼──> Sign Collisions &
                      └──> Task Vector 3 (τ₃ = θ₃ - θ₀) ──┘    Parameter Smearing
───────────────────────────────────────────────────────────────────
TIES / DARE Pipeline:
                      ┌──> Sparsification (Top-p / Bernoulli Drop)
                      ├──> Sign Consensus Election
                      └──> Disjoint Redundancy Elimination ──> Stable Merged Model

The Mechanics of Weight Interference

Why does naive addition of task vectors fail in large language models? Yadav et al. (2023) categorized the failure modes into two primary phenomena:

1. Redundant Parameter Accumulation: A vast majority of parameter changes across fine-tuned models represent small, noise-level drifts that do not contribute meaningfully to task specialization. Accumulating these low-magnitude deltas across multiple models shifts the baseline weights away from the pre-trained manifold, degrading core reasoning and language modeling perplexity.
2. Sign Disagreement and Destructive Interference: When different tasks pull the same parameter in opposing directions (e.g., $\tau_{1,i} > 0$ while $\tau_{2,i} < 0$), linear addition causes mutual cancellation:

$$\tau_{merged, i} = \lambda_1 \tau_{1,i} + \lambda_2 \tau_{2,i} \approx 0$$

This cancellation nullifies the learned features for both tasks. Conversely, when large opposing updates are averaged, the resulting value matches neither task's optimal representation.

Advanced Merging Methodologies

To resolve parameter interference, researchers developed non-linear geometric interpolation and sparsification-based consensus algorithms.

1. Spherical Linear Interpolation (SLERP)

Linear interpolation between two high-dimensional weight vectors $\theta_1$ and $\theta_2$ cuts through the interior of the hypersphere, reducing the norm of the intermediate vector. In deep neural networks, maintaining weight vector magnitude is critical for preserving activation variance across normalization layers.

SLERP interpolates vectors along the spherical manifold at constant angular velocity:

$$\theta_{SLERP}(t) = \frac{\sin((1-t)\Omega)}{\sin\Omega} \theta_1 + \frac{\sin(t\Omega)}{\sin\Omega} \theta_2$$

where $\Omega = \arccos\left(\frac{\theta_1 \cdot \theta_2}{\|\theta_1\| \|\theta_2\|}\right)$ is the angle between the vectors, and $t \in [0, 1]$ is the interpolation factor.

• Strengths: Preserves geometric norm and representation geometry; highly effective for fusing two closely related models (e.g., an instruction fine-tune and a reasoning fine-tune of the same base).
• Limitations: Inherently pairwise; extending SLERP to $N > 2$ models requires hierarchical tree reduction or spherical barycentric approximations.

2. TIES-Merging (Trim, Elect Sign, and Merge)

Proposed by Yadav et al. (2023), TIES-Merging resolves parameter interference through a deterministic three-stage pipeline:

1. Trim: For each task vector $\tau_k$, retain only the top $p\%$ parameters with the largest absolute magnitude, setting the remaining $(100 - p)\%$ to zero:

$$\hat{\tau}_k = \text{Trim}(\tau_k, p)$$

1. Elect Sign: Compute the aggregate sign consensus vector $\gamma \in \{-1, +1\}^d$ across all trimmed task vectors based on total directional magnitude:

$$\gamma_i = \text{sgn}\left(\sum_{k=1}^K \hat{\tau}_{k, i}\right)$$

1. Disjoint Merge: For each parameter index $i$, average only the updates from models whose sign matches the elected consensus $\gamma_i$, ignoring models that disagree:

$$\tau_{TIES, i} = \frac{1}{|A_i|} \sum_{k \in A_i} \hat{\tau}_{k, i}, \quad \text{where } A_i = \{k \mid \text{sgn}(\hat{\tau}_{k, i}) = \gamma_i\}$$ The final merged model is: $$\theta_{merged} = \theta_0 + \lambda \tau_{TIES}$$

3. DARE (Drop And REscale)

Yu et al. (2023) introduced DARE, demonstrating that up to 90% to 99% of delta parameters in fine-tuned LLMs can be dropped entirely without performance degradation, provided the surviving weights are appropriately rescaled.

For each task vector $\tau_k$, DARE applies a stochastic Bernoulli mask $m_k \sim \text{Bernoulli}(1 - p)$, where $p$ is the drop rate (typically $0.7 \le p \le 0.99$), and scales the remaining parameters by $\frac{1}{1 - p}$:

$$\tilde{\tau}_k = \frac{1}{1 - p} \left(m_k \odot \tau_k\right)$$

DARE acts as a sparsification operator that can be paired with either linear task arithmetic (DARE-Linear) or sign consensus (DARE-TIES). By eliminating over 90% of parameter deltas per model, DARE minimizes spatial collisions when merging 5 to 20+ specialized models into a single base checkpoint.

4. Passthrough (Frankenmerging)

Unlike weight interpolation, Passthrough merges (Goddard et al., 2024) concatenate layers along the depth dimension. For example, taking layers 0-16 from Model A, layers 8-24 from Model B, and layers 16-32 from Model C to construct an expanded architecture. While capable of creating intermediate-sized models (e.g., expanding an 8B model into an 11B or 14B variant), frankenmerging requires careful layer index alignment to prevent residual connection disharmony.

Architectural Trade-Off Breakdown

• Model Soup / LERP: Linear parameter averaging across two or more models. No parameter sparsification or sign conflict handling. Best suited for ensembling multiple fine-tuning runs on the identical task.
• Task Arithmetic: Direct additive combinations of task delta vectors ($\tau_k = \theta_k - \theta_0$). Retains all weights without sparsification or sign election. Effective for low-conflict task transfer and targeted capability negation.
• SLERP (Spherical Linear Interpolation): Constant angular velocity rotation between two models preserving geometric vector norms. Pairwise only; optimal for blending a base model and an instruction-tuned checkpoint.
• TIES-Merging: Magnitude-based quantile trimming (top-p%), sign consensus election, and disjoint parameter averaging. Specifically designed for multi-task fusion where fine-tuned models exhibit directional gradient conflicts.
• DARE-TIES: High-rate Bernoulli dropout (70% to 99% parameter elimination) with weight rescaling followed by TIES sign consensus. Optimal for scaling model fusion across numerous (3 to 20+) domain-specific checkpoints while preventing representation collapse.
• Passthrough (Frankenmerging): Concatenates layers across depth dimensions without weight interpolation. Used for expanding layer depth and intermediate capacity scaling.

Systems Implementation: MergeKit Out-of-Core Processing

Executing model merges on frontier weights (e.g., Llama-3-70B, Qwen-2.5-72B, or Mixtral-8x22B) presents substantial memory challenges. Loading multiple 70B parameter models in 16-bit precision simultaneously would require upwards of 400 GB to 600 GB of VRAM.

The MergeKit architecture addresses this via an out-of-core, tensor-by-tensor execution pipeline:

MergeKit Out-of-Core Execution Flow
┌────────────────────────────────────────────────────────┐
│ Safetensors Shards on NVMe (Model A, Model B, Base)   │
└──────────────────────────┬─────────────────────────────┘
                           │ Memory-Mapped Tensor Stream
                           ▼
┌────────────────────────────────────────────────────────┐
│ RAM / CPU Cache Workspace (Single Tensor Slice: ~200MB)│
│  1. Compute Task Deltas (τ_A, τ_B)                    │
│  2. Apply DARE Sparsification & Scaling                │
│  3. Execute Sign Election & Disjoint Masking           │
│  4. Add to Base Tensor (θ₀)                            │
└──────────────────────────┬─────────────────────────────┘
                           │ Streamed Output
                           ▼
┌────────────────────────────────────────────────────────┐
│ Serialized Safetensors on Disk (Target Checkpoint)     │
└────────────────────────────────────────────────────────┘

1. Lazy Tensor Iteration: Weights are read iteratively from disk via memory-mapped I/O (mmap). Only the current parameter tensor (e.g., model.layers.14.self_attn.q_proj.weight) across all input models is loaded into system memory.
2. Device Agnostic Execution: Tensor operations run on CPU RAM or a single consumer GPU. Merging a 70B model requires less than 32 GB of system RAM and zero multi-GPU clusters.
3. Graph Execution & Architecture Parsing: MergeKit maps layer names across heterogeneous naming conventions (e.g., translating between different fine-tuning framework naming schemes) and executes the specified recipe sequentially.

Production Recipe Configuration Example

Below is a production MergeKit YAML configuration implementing a DARE-TIES merge of a base model with coding, mathematical, and safety instruction fine-tunes:

models:
  - model: meta-llama/Llama-3.1-8B
    # Base model acts as reference origin
  - model: cognitivecomputations/dolphin-2.9-llama3-8b
    parameters:
      density: 0.65
      weight: 0.35
  - model: TechxGenus/Meta-Llama-3.1-8B-Instruct-AWQ
    parameters:
      density: 0.70
      weight: 0.35
  - model: deepseek-ai/DeepSeek-Coder-V2-Lite-Base
    parameters:
      density: 0.50
      weight: 0.30
merge_method: dare_ties
base_model: meta-llama/Llama-3.1-8B
parameters:
  int8_mask: true
dtype: bfloat16

Production Serving Economics: Merged Models vs. Routing and Multi-LoRA

Deploying merged models introduces direct architectural trade-offs against multi-adapter routing and disaggregated multi-endpoint architectures:

Serving Architecture Comparison
─────────────────────────────────────────────────────────────────────────
1. Disaggregated Multi-Endpoint:
   [Gateway Router] ──> [Endpoint 1: Code 70B]  (4x H100s)
                    ──> [Endpoint 2: Math 70B]  (4x H100s)
                    ──> [Endpoint 3: Chat 70B]  (4x H100s)
   Total Infrastructure: 12x H100 GPUs. Low KV-cache reuse across tasks.

2. Dynamic Multi-LoRA Serving (S-LoRA / Punica):
   [Gateway Router] ──> [Unified Engine: Base 70B + Dynamic LoRAs] (4x H100s)
   Trade-off: Kernel multiplexing overhead, memory fragmentation on adapters.

3. Static Merged Model:
   [Gateway Router] ──> [Unified Engine: Merged 70B] (4x H100s)
   Trade-off: Standard dense inference kernels, 100% KV-cache sharing, 
              zero adapter-switching latency overhead.
─────────────────────────────────────────────────────────────────────────

1. KV Cache Sharing and Batching Efficiency

In multi-turn agent workflows where tasks alternate between code synthesis, tool execution, and dialogue summary, routing requests to separate endpoints invalidates KV caches. A single merged model processes all steps within the same inference context, maximizing prefix cache hits and continuous batching throughput.

2. Elimination of Kernel Overhead

Multi-LoRA frameworks require segmented matrix multiplications (batched GEMM) to apply varying adapter weights per request in a batch. A merged model produces a single dense weight matrix, allowing standard high-throughput tensor-parallel kernels (e.g., FlashAttention-3, vLLM CUDA graph runners) to operate at peak hardware Model FLOPs Utilization (MFU).

3. VRAM Footprint

Serving three distinct 70B models in FP8 or 16-bit precision requires three dedicated GPU clusters (e.g., $3 \times 4 \times \text{H100}$). A merged checkpoint serves all three domain competencies within a single $4 \times \text{H100}$ node, reducing idle infrastructure costs by up to 66%.

Production Pitfalls and Validation Protocols

Model merging is not a guaranteed lossless operation. In production deployments, merged checkpoints frequently encounter specific failure modes:

1. Safety and Refusal Degradation: Alignment fine-tuning (RLHF/DPO) typically introduces sparse, fragile weight adjustments. Merging an aligned model with an aggressive coding or raw domain fine-tune often overwrites safety guardrails. Teams must re-verify adversarial jailbreak benchmarks (e.g., StrongREJECT, WildJailbreak) post-merge.
2. Tokenizer Vocabulary Divergence: Merging models with differing vocabulary sizes (or modified embedding matrices) corrupts token-to-embedding mappings. Input models must share an identical tokenizer and embedding dimension unless explicit vocabulary alignment layers are applied.
3. Quantization Sensitivity: Merged weights often exhibit higher activation kurtosis and outlier parameter distributions than standard pre-trained weights. Standard post-training quantization (PTQ) schemes (e.g., FP8, AWQ, or GPTQ) can suffer elevated perplexity degradation on merged models unless calibration datasets incorporate data from all merged domain distributions.
4. Attention Imbalance in QK/VO Projections: When task vectors disproportionately modify Query-Key projection matrices relative to Value-Output matrices, attention scores can saturate, leading to repetitive generation loops or degraded long-context retrieval performance.

Operational Summary

Model merging transforms multi-task LLM adaptation from an expensive distributed training challenge into a deterministic parameter-space engineering process.

For pairwise combinations of base and instruction models, SLERP remains the optimal choice for preserving weight norms. For multi-model integration across 3 to 10+ domain-specific fine-tunes, DARE-TIES provides the highest stability by aggressively pruning noise parameters and resolving directional sign conflicts.

By integrating tools like MergeKit into automated CI/CD evaluation pipelines, teams can continuously synthesize specialized domain updates into unified, cost-efficient production models.

Sources

• Editing Models with Task Arithmetic (Ilharco et al., 2022)
• Resolving Interference When Merging Models (Yadav et al., 2023 - TIES-Merging)
• Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch (Yu et al., 2023 - DARE)
• Arcee's MergeKit: A Toolkit for Merging Large Language Models (Goddard et al., 2024)
• Model Soups: Averaging Weights of Multiple Fine-Tuned Models Improves Accuracy Without Increasing Inference Time (Wortsman et al., 2022)
• Animating Rotation with Quaternion Curves (Shoemake, 1985 - SLERP)