BitNet b1.58: Mathematical Foundations of 1.58-Bit Ternary Quantization, Absmean Scaling, and Multiplication-Free LLM Architectures

Standard Large Language Models (LLMs) rely on 16-bit floating-point (FP16 or BF16) matrix multiplications for both pre-training and autoregressive inference. While post-training quantization techniques (such as GPTQ, AWQ, and dynamic FP8/INT4 scaling) compress weights after training, they often introduce perplexity degradation at ultra-low bitwidths ($<4$ bits) or still require floating-point dequantization at runtime.

Quantization-aware pre-training architectures directly tackle this computational boundary. In The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits, researchers introduced BitNet b1.58, an architectural paradigm where every linear projection weight is constrained to ternary values: $\{-1, 0, +1\}$. By replacing floating-point multiply-accumulate (MAC) operations with pure integer additions, BitNet b1.58 fundamentally alters the arithmetic intensity, energy profile, and memory footprint of neural language models without sacrificing perplexity or downstream task performance.

1. Information-Theoretic Foundations: Why 1.58 Bits?

Standard binary neural networks (1-bit architectures like BitNet b1) constrain weights strictly to binary states:

$$\mathcal{W}_{\text{binary}} \in \{-1, +1\}$$

In a binary representation, each parameter encodes $\log_2(2) = 1.00\text{ bit}$ of information. However, binary models struggle with feature filtering because every parameter must actively exert either a positive or negative influence on the activation stream. A binary weight cannot represent an inactive or null connection without balancing opposite activations.

BitNet b1.58 expands the weight space to a ternary alphabet:

$$\mathcal{W}_{\text{ternary}} \in \{-1, 0, +1\}$$

From Shannon information theory, the maximum entropy or theoretical information capacity per parameter for three discrete states is:

$$H = \log_2(3) \approx 1.58496\text{ bits}$$

The inclusion of the explicit $0$ state provides three critical mathematical properties:

1. Native Feature Filtering: The model can explicitly assign zero weight to irrelevant activations, providing structured sparsity and gating capabilities directly in the weight matrix.
2. Preservation of Gradient Dynamics: The zero state prevents the optimization trajectory from oscillating violently between $+1$ and $-1$ across gradient updates.
3. Information Density Matching: Empirical evaluations demonstrate that at $1.58$ bits per parameter, model capacity crosses the threshold required to match full-precision Transformer perplexity at equal token counts and model widths.

2. Mathematical Formulation of the BitLinear b1.58 Layer

In standard Transformers, linear projections take the form $Y = XW^T$, where $X \in \mathbb{R}^{B \times L \times d_{\text{in}}}$ and $W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$. In BitNet b1.58, standard linear layers are replaced by BitLinear modules that quantize both weights and intermediate activations.

Weight Quantization: Absmean Scaling

Weights $W$ are quantized to ternary values using an absmean (mean absolute value) scaling factor $\gamma$. For a weight tensor $W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$:

$$\gamma = \frac{1}{d_{\text{out}} d_{\text{in}}} \sum_{i=1}^{d_{\text{out}}} \sum_{j=1}^{d_{\text{in}}} |W_{ij}|$$

Each continuous latent weight is scaled by $\gamma$ and mapped to $\{-1, 0, +1\}$ using a rounded clamping operator:

$$\widetilde{W}_{ij} = \text{RoundClip}\left(\frac{W_{ij}}{\gamma + \epsilon}, -1, 1\right)$$

Where $\text{RoundClip}(z, a, b) = \max(a, \min(b, \lfloor z + 0.5 \rfloor))$ and $\epsilon = 10^{-5}$ prevents division by zero.

Activation Quantization: Absmax Scaling

Before linear transformation, intermediate activations are normalized via RMSNorm to zero-center the representation and remove scale variance. Activations are subsequently quantized to an 8-bit signed integer range (INT8), yielding $Q_b = 2^{k-1} = 128$ levels ($[-128, 127]$ or symmetric $[-127, 127]$):

$$\beta = \max_{i, j} |X_{ij}|$$

$$\widetilde{X}_{ij} = \text{RoundClip}\left( X_{ij} \times \frac{Q_b - 1}{\beta + \epsilon}, -(Q_b - 1), Q_b - 1 \right)$$

For cross-batch numerical stability, $\beta$ can be computed on a per-token basis (token-wise absmax) or per-tensor basis.

The Linear Transformation and Rescaling

The resulting matrix multiplication between quantized activations $\widetilde{X}$ and quantized ternary weights $\widetilde{W}$ is computed and rescaled back to the continuous range:

$$Y = \left( \widetilde{X} \widetilde{W}^T \right) \times \frac{\beta \gamma}{Q_b - 1}$$

Because $\widetilde{W}_{ij} \in \{-1, 0, +1\}$, the core computation $\widetilde{X} \widetilde{W}^T$ requires zero floating-point or integer multiplications. It reduces entirely to additions and subtractions of INT8 activations into INT32 accumulators.

3. Training Dynamics and the Straight-Through Estimator (STE)

Directly optimizing discrete parameters via gradient descent is impossible because the derivative of $\text{RoundClip}(z, -1, 1)$ with respect to continuous input $z$ is zero almost everywhere and undefined at discontinuity boundaries:

$$\frac{\partial \text{RoundClip}(z)}{\partial z} = 0 \quad \forall z \notin \mathbb{Z} + 0.5$$

To enable end-to-end backpropagation, BitNet utilizes the Straight-Through Estimator (STE), originally formalized by Bengio et al. (2013).

Latent Continuous Weights and Gradient Flow

During training:

1. The model maintains full-precision latent weights $W_{\text{latent}} \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$.
2. In the forward pass, $W_{\text{latent}}$ is quantized to ternary $\widetilde{W}$ via absmean scaling.
3. In the backward pass, the discrete rounding operator is bypassed in the computation graph:

$$\frac{\partial \mathcal{L}}{\partial W_{\text{latent}}} \approx \frac{\partial \mathcal{L}}{\partial \widetilde{W}}$$

1. The optimizer (AdamW or standard SGD) accumulates continuous gradients and updates $W_{\text{latent}}$ directly:

$$W_{\text{latent}}^{(t+1)} = W_{\text{latent}}^{(t)} - \eta \cdot \frac{\widehat{m}_t}{\sqrt{\widehat{v}_t} + \epsilon}$$

Because the continuous latent weights accumulate minute gradient steps across training batches, parameters can smoothly transition across quantization thresholds (e.g. from 0 to +1 or -1).

4. Hardware Arithmetic and Energy Economics

The shift from 16-bit floating-point arithmetic to ternary-weight integer arithmetic alters the energy efficiency of deep learning workloads.

Multiplication vs. Addition Energy on Silicon

In standard CMOS semiconductor manufacturing (e.g., 7nm / 5nm processes), floating-point arithmetic requires complex multi-stage barrel shifters, exponent alignment logic, and mantissa multiplication arrays. Integer addition circuits require minimal transistor area and low switching capacitance.

| Operation | Format | Energy per Operation (pJ) | Relative Energy | | :--- | :--- | :--- | :--- | | Multiplication | FP32 | ~3.7 pJ | 123.3x | | Multiplication | FP16 / BF16 | ~1.1 pJ | 36.7x | | Addition | FP32 | ~0.9 pJ | 30.0x | | Addition | FP16 / BF16 | ~0.4 pJ | 13.3x | | Multiplication | INT8 | ~0.2 pJ | 6.7x | | Addition | INT8 / INT32 Accum | ~0.03 pJ | 1.0x |

Because BitNet b1.58 transforms matrix operations into purely INT8 addition, the computational energy consumed during GEMV/GEMM matrix kernels drops by over an order of magnitude compared to FP16 baselines.

Memory Bandwidth and Latency Bounds

Autoregressive LLM generation at small batch sizes is fundamentally memory-bandwidth bound. The latency to generate token $t$ is governed by the time required to stream weights from GPU High Bandwidth Memory (HBM) or CPU DRAM into SRAM/registers:

$$T_{\text{step}} \approx \frac{\text{Model Parameters (Bytes)}}{\text{Memory Bandwidth (Bytes/s)}}$$

By packing ternary values (e.g., 5 ternary values per 8-bit byte, since $3^5 = 243 < 256$), BitNet b1.58 reduces weight storage from 16 bits per parameter to under 1.6 bits. This delivers an approximate $7\times$ to $8\times$ reduction in model parameter byte volume, translating to near-linear throughput increases on memory-constrained hardware.

5. Architectural Scaling and Empirical Comparisons

Empirical scaling laws detailed by Ma et al. (2024) demonstrate that the performance gap between ternary models and full-precision Transformer baselines narrows as parameter scale increases.

Scaling Comparison Across Parameter Regimes

At smaller scales (e.g., 700M parameters), full-precision models maintain a slight perplexity advantage due to the absolute parameter budget. However, starting at 3B parameters, BitNet b1.58 matches or exceeds the zero-shot accuracy and perplexity of open FP16 baselines (such as LLaMA-3B) trained on identical token volumes.

| Model Scale | Weight Precision | Memory Footprint (Weights) | Arithmetic Kernel Type | Relative Inference Speed | | :--- | :--- | :--- | :--- | :--- | | LLaMA Baseline (3B) | FP16 (16-bit) | 6.0 GB | FP16 GEMM (MAC) | 1.00x | | BitNet b1.58 (3B) | Ternary (1.58-bit) | 0.8 GB | INT8 Addition (Add-only) | 2.71x | | LLaMA Baseline (70B) | FP16 (16-bit) | 140.0 GB | FP16 GEMM (MAC) | 1.00x | | BitNet b1.58 (70B) | Ternary (1.58-bit) | 17.8 GB | INT8 Addition (Add-only) | 8.90x |

At 70B parameter scale, the memory reduction allows an entire model to reside in a single 24GB or 32GB memory pool (such as standard consumer GPUs or edge accelerators), eliminating multi-GPU tensor-parallel communication overheads.

6. Implementation Considerations and Inference Engines

Deploying BitNet b1.58 models in production requires custom inference kernels tailored to ternary decoding rather than standard cuBLAS / CUTLASS routines.

Vectorized Ternary GEMV with SIMD

On x86 and ARM architectures, specialized inference engines like bitnet.cpp implement customized SIMD instructions (AVX2, AVX-512, and ARM NEON):

1. Weight Unpacking: Packed 2-bit or 1.58-bit ternary registers are unpacked into sign masks and zero masks.
2. Masked Addition: Rather than executing hardware multiplications, vector instructions use sign masks to conditionally negate INT8 activations and zero masks to skip inactive elements:

$$\text{accumulator} \mathrel{+}= (X_{\text{int8}} \oplus \text{sign}) - \text{sign} \quad \text{where } W \ne 0$$

1. Scaling Factor Fusion: The scalar rescaling $\frac{\beta \gamma}{Q_b - 1}$ is applied once per block or per row upon accumulating the final INT32 vector, amortizing the single floating-point multiplication across hundreds of dimensions.

Sources

• Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Bhatia, S., Ding, R., Xue, J., & Wei, F. (2024). The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits. arXiv:2402.17764.
• Wang, H., Ma, S., Dong, L., Huang, S., Wang, H., Ma, L., Yang, F., Wang, R., Wu, Y., & Wei, F. (2023). BitNet: Scaling 1-bit Transformers for Large Language Models. arXiv:2310.11453.
• Bengio, Y., Léonard, N., & Courville, A. (2013). Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. arXiv:1308.3432.
• Zhang, B., & Sennrich, R. (2019). Root Mean Square Layer Normalization. Advances in Neural Information Processing Systems (NeurIPS 2019).
• Loshchilov, I., & Hutter, F. (2019). Decoupled Weight Decay Regularization. International Conference on Learning Representations (ICLR 2019).