llms.blog
Model Merging3 articles

Model Merging

Articles

  • Model Merging and Task Arithmetic: Mathematical Foundations, Task Vectors, TIES-Merging, DARE, and Spherical Linear Interpolation (SLERP)

    Model Merging and Task Arithmetic: Mathematical Foundations, Task Vectors, TIES-Merging, DARE, and Spherical Linear Interpolation (SLERP) Model merging enables the combination of multiple specialized large language models into a unified checkpoint without retraining, backpropagation, or access to the original training datasets. Rather than running costly multi-task supervised fine-tuning across disparate domains, practitioners can directly manipulate parameter tensors in weight space to fuse ca

    1 min

  • Model Merging in Weight Space: Mathematical Foundations of Task Vectors, TIES-Merging, DARE, and Spherical Linear Interpolation

    Training large language models across multiple downstream domains typically requires expensive multi-task supervised fine-tuning (SFT) or sequential continual learning. Both paths present steep practical challenges: multi-task training demands simultaneous access to massive, curated datasets and massive compute budgets, while sequential fine-tuning suffers from catastrophic forgetting. Model merging has emerged as a data-free, compute-efficient alternative. By combining the parameters of multip

    1 min

  • Model Merging in Large Language Models: How Task Arithmetic, TIES, and DARE Combine Checkpoints Without Training

    Fine-tuning foundation models for specialized tasks typically produces isolated checkpoints. A model adapted for mathematical reasoning retains high numerical precision but often degrades in general dialogue or code generation. Traditionally, unifying these capabilities required multi-task training: gathering mixed datasets, re-running optimization across multiple GPUs, and managing gradient conflicts during backpropagation. Model merging provides an alternative paradigm. By operating directly

    1 min