llms.blog
AI

Liquid AI Releases Quantization-Aware Distilled Q4_0 Checkpoints for LFM2.5 Models

Liquid AI has released Quantization-Aware Distillation (QAD) Q4_0 GGUF checkpoints for its LFM2.5 model series, allowing edge runtimes to execute 4-bit quantized non-transformer architectures without the accuracy degradation typically associated with standard post-training quantization (PTQ). The release covers four models in the LFM2.5 family: LFM2.5-230M, LFM2.5-350M, LFM2.5-1.2B-Instruct, and LFM2.5-2.6B. The checkpoints are packaged in the standard GGUF format and run across llama.cpp and c

axc
axc
2 min
Liquid AI Releases Quantization-Aware Distilled Q4_0 Checkpoints for LFM2.5 Models

Liquid AI has released Quantization-Aware Distillation (QAD) Q4_0 GGUF checkpoints for its LFM2.5 model series, allowing edge runtimes to execute 4-bit quantized non-transformer architectures without the accuracy degradation typically associated with standard post-training quantization (PTQ).

The release covers four models in the LFM2.5 family: LFM2.5-230M, LFM2.5-350M, LFM2.5-1.2B-Instruct, and LFM2.5-2.6B. The checkpoints are packaged in the standard GGUF format and run across llama.cpp and compatible inference backends.

Quantization-Aware Distillation Architecture

Benchmark Recovery and Accuracy Retention

Traditional post-training quantization methods, such as standard rounding or calibration-based PTQ, often introduce noticeable performance penalties when compressing models below 3 billion parameters down to 4 bits. Liquid AI addressed this by combining knowledge distillation directly with quantization objectives during post-training.

Evaluated across a benchmark suite spanning reasoning (GPQA Diamond, MMLU-Pro), instruction following (IFEval, IFBench, Multi-IF), tool calling (BFCLv4), and mathematics (GSM8K for sub-1B models, AIME25 for 1.2B and 2.6B models), the QAD Q4_0 checkpoints retained near-uncompressed capability:

  • LFM2.5-230M: Retained 97.1% of original BF16 baseline performance.
  • LFM2.5-350M: Retained 96.5% of original BF16 baseline performance.
  • LFM2.5-1.2B-Instruct: Retained 97.4% of original BF16 baseline performance.
  • LFM2.5-2.6B: Retained 96.6% of original BF16 baseline performance.

In head-to-head comparisons against standard post-training quantization levels, the 230M and 350M QAD Q4_0 models matched the accuracy of Q5_K_M checkpoints while running at significantly smaller memory footprints. The 1.2B and 2.6B models matched Q4_K_M and Unsloth UD-Q4_K_XL quality thresholds.

Hardware Profiling and Edge Throughput

Liquid AI benchmarked decode throughput across four edge deployment targets representing different compute architectures: Apple MacBook Pro (Metal GPU), Minisforum NucBox EVO-X2 (integrated GPU), Samsung Galaxy S26 Ultra (Arm CPU), and Raspberry Pi 5 (Arm CPU).

Because standard Q4_0 formats allow simplified vector dequantization compared to complex block-level k-quantization schemes (such as Q4_K_M or Q5_K_M), the QAD checkpoints delivered measurable throughput advantages:

  • Sub-1B Tier (230M and 350M): Achieved 4% to 33% higher token decode throughput compared to Q5_K_M while maintaining equivalent evaluation accuracy.
  • Mid-Tier (1.2B and 2.6B): Achieved 3% to 14% higher decode throughput relative to Q4_K_M formats.

Edge Runtime Integration

The updated models are available on Hugging Face under the LiquidAI organization repository. Developers can run the QAD checkpoints directly via llama-cli or downstream runtimes without specialized custom kernels, making low-latency local agent execution viable on low-power consumer devices and embedded single-board computers.

Sources

AIOpen WeightsEdge AIQuantization
ShareX (Twitter)LinkedIn

Written by

axc

More to read

  • Grokking in Large Language Models: How Weight Decay and Circuit Efficiency Drive Delayed Generalization

    Grokking in Large Language Models: How Weight Decay and Circuit Efficiency Drive Delayed Generalization In standard machine learning paradigms, model generalization closely tracks training loss: as an optimizer minimizes loss on training data, performance on held-out validation data improves in tandem until the model begins to overfit. In 2022, researchers at OpenAI observed a phenomenon that inverted this assumption: small neural networks trained on algorithmic tasks achieved near-zero trainin

    1 min

  • Serverless GPU Inference in Production: Cold Starts, GPU Memory Snapshotting, and Weight Paging Architectures

    Serverless GPU Inference in Production: Cold Starts, GPU Memory Snapshotting, and Weight Paging Architectures Deploying large language models on dedicated cloud GPUs creates an uncomfortable financial trade-off: keeping enterprise accelerators such as NVIDIA H100s or A100s warm 24/7 costs thousands of dollars per instance each month, yet scaling instances to zero introduces severe latency penalties. When traffic arrives at a dormant node, a standard inference server cold start can take anywhere

    1 min

  • Transformer Feed-Forward Networks as Key-Value Memories: How First-Layer Keys and Second-Layer Values Store Knowledge

    Transformer Feed-Forward Networks as Key-Value Memories: How First-Layer Keys and Second-Layer Values Store Knowledge In modern autoregressive Transformers, the division of labor between attention heads and multi-layer perceptron (MLP) blocks is often summarized through a clean functional split: attention routes information across sequence positions, while feed-forward networks (FFNs) process information per position. Yet for years, the exact mechanism by which FFNs process that information rem

    1 min