Rotary Position Embeddings: Mathematical Foundations, Complex Rotations, and Long-Context Scaling

Standard transformer architectures lack an intrinsic mechanism to model sequence order. Because the self-attention operation is permutation-equivariant, shuffling the input token sequence produces an identical permutation in the output representations unless positional signals are explicitly injected.

Early architectures addressed this constraint through additive position embeddings, either via fixed sinusoidal functions or learnable absolute position vectors. However, additive absolute encodings struggle with sequence generalization and do not naturally encode relative distances between tokens.

Rotary Position Embedding (RoPE), introduced by Su et al. in RoFormer: Enhanced Transformer with Rotary Position Embedding (2021), resolved this structural limitation. By encoding positional information through orthogonal rotations in complex vector space, RoPE injects relative positional dependency directly into the attention inner product while retaining absolute token-level representation. RoPE has become the default positional encoding standard across modern autoregressive models, including LLaMA, Mistral, Gemma, Qwen, and DeepSeek.

The Limitations of Classical Positional Encodings

In the original transformer architecture described by Vaswani et al. in Attention Is All You Need (2017), positional information was introduced by adding sinusoidal vectors directly to token embeddings:

$$x_m = e_m + p_m$$

where $e_m$ is the token embedding at index $m$, and $p_m$ is the absolute positional vector composed of interleaved sine and cosine frequencies:

$$p_{m, 2i} = \sin\left(\frac{m}{10000^{2i/d}}\right), \quad p_{m, 2i+1} = \cos\left(\frac{m}{10000^{2i/d}}\right)$$

While this enables linear transformations to learn linear shifts, the dot product between query $q_m = W_q x_m$ and key $k_n = W_k x_n$ expands into four additive terms:

$$q_m^T k_n = (W_q e_m + W_q p_m)^T (W_k e_n + W_k p_n) = e_m^T W_q^T W_k e_n + e_m^T W_q^T W_k p_n + p_m^T W_q^T W_k e_n + p_m^T W_q^T W_k p_n$$

This expansion mixes token content and absolute positions across cross-terms. The attention score is not a pure function of relative distance $m - n$, and the additive position vector corrupts semantic representations in the lower layers.

Subsequent research introduced relative positional encodings. Shaw et al. in Self-Attention with Relative Position Representations (2018), Transformer-XL by Dai et al. in Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context (2019), and T5 by Raffel et al. in Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2020) introduced learnable bias terms directly into the attention score matrix:

$$A_{m, n} = q_m^T k_n + b_{m-n}$$

While effective at capturing relative distances, matrix-level relative biases have significant disadvantages:

• They require storing or generating an $L \times L$ relative bias matrix, increasing memory bandwidth requirements.
• They are incompatible with linear attention formulations and require customized modifications to hardware-accelerated kernels such as FlashAttention.
• They do not directly modify token representations before attention computation.

Mathematical Formulation of RoPE

Su et al. formulated positional encoding as a constraint optimization problem. Given a query vector $q_m$ at sequence index $m$ and a key vector $k_n$ at sequence index $n$, the objective is to find transformation functions $f_q(x_m, m)$ and $f_k(x_n, n)$ such that their inner product depends exclusively on the token contents and the relative distance $m - n$:

$$\langle f_q(x_m, m), f_k(x_n, n) \rangle = g(x_m, x_n, m - n)$$

2D Complex Space Derivation

To solve this functional equation, consider a 2D vector space. Any 2D real vector $x = (x^{(1)}, x^{(2)})^T$ can be represented as a complex number:

$$x = x^{(1)} + i x^{(2)} = r e^{i\phi}$$

Applying a rotation proportional to position $m$ with angular frequency $\theta$ corresponds to complex multiplication:

$$f_q(x_m, m) = q_m e^{i m \theta}$$ $$f_k(x_n, n) = k_n e^{i n \theta}$$

Computing the inner product of two complex numbers corresponds to taking the real part of $z_1 z_2^$, where $$ denotes the complex conjugate:

$$\langle f_q(x_m, m), f_k(x_n, n) \rangle = \text{Re}\left( (q_m e^{i m \theta}) (k_n e^{i n \theta})^* \right)$$ $$\langle f_q(x_m, m), f_k(x_n, n) \rangle = \text{Re}\left( q_m k_n^* e^{i (m - n) \theta} \right)$$

The absolute position coordinates $m$ and $n$ cancel out, leaving only the relative displacement $m - n$. In matrix form over $\mathbb{R}^2$, this transformation is expressed as an orthogonal 2D rotation matrix:

$$R_{\theta, m} = \begin{pmatrix} \cos(m\theta) & -\sin(m\theta) \\ \sin(m\theta) & \cos(m\theta) \end{pmatrix}$$

The inner product satisfies:

$$(R_{\theta, m} q)^T (R_{\theta, n} k) = q^T R_{\theta, m}^T R_{\theta, n} k = q^T R_{\theta, n - m} k$$

because $R_{\theta, m}^T = R_{\theta, -m}$ and $R_{\theta, -m} R_{\theta, n} = R_{\theta, n - m}$.

Multi-Dimensional Generalization

For a $d$-dimensional head embedding space (where $d$ is even), $\mathbb{R}^d$ is decomposed into $d/2$ orthogonal 2D subspaces. Each subspace is assigned a distinct angular frequency $\theta_i$:

$$\theta_i = \theta_{\text{base}}^{-2(i - 1)/d}, \quad i \in \{1, 2, \dots, d/2\}$$

The canonical value for the base frequency is $\theta_{\text{base}} = 10000$.

The full $d$-dimensional rotation matrix $R_{\Theta, m}^d$ is a block-diagonal matrix:

$$R_{\Theta, m}^d = \begin{pmatrix} R_{\theta_1, m} & 0 & \dots & 0 \\ 0 & R_{\theta_2, m} & \dots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \dots & R_{\theta_{d/2}, m} \end{pmatrix}$$

Because $R_{\Theta, m}^d$ is orthogonal, it preserves vector norms: $\|R_{\Theta, m}^d x\| = \|x\|$. Positional encoding via RoPE modifies only the direction of the vector, preserving the magnitude of query and key states.

Computational Implementation and Efficiency

Explicitly constructing and multiplying $d \times d$ rotation matrices is computationally wasteful. In practice, RoPE is implemented via elementwise vector operations.

Given an input vector $x \in \mathbb{R}^d$, the vector is split into even and odd index pairs, or sliced into two halves:

$$\tilde{x} = \begin{pmatrix} -x_2 \\ x_1 \\ -x_4 \\ x_3 \\ \vdots \\ -x_d \\ x_{d-1} \end{pmatrix}$$

The rotated vector is computed directly as:

$$R_{\Theta, m}^d x = x \odot \cos(m\Theta) + \tilde{x} \odot \sin(m\Theta)$$

where $\cos(m\Theta)$ and $\sin(m\Theta)$ are precomputed frequency vectors duplicated across 2D pairs.

Reference PyTorch Implementation

import torch

def precompute_rope_frequencies(dim: int, seq_len: int, theta_base: float = 10000.0) -> tuple[torch.Tensor, torch.Tensor]:
    """Precomputes cosine and sine frequency tables for RoPE."""
    # dim must be even
    channel_indices = torch.arange(0, dim, 2).float()
    theta = 1.0 / (theta_base ** (channel_indices / dim))
    positions = torch.arange(seq_len).float()
    
    # Outer product: [seq_len, dim / 2]
    angles = torch.outer(positions, theta)
    
    # Repeat along the last dimension to match [seq_len, dim]
    angles = torch.repeat_interleave(angles, 2, dim=-1)
    return torch.cos(angles), torch.sin(angles)

def apply_rotary_emb(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
    """Applies RoPE rotation to input tensor x of shape [batch, heads, seq_len, head_dim]."""
    # Create [-x_1, x_0, -x_3, x_2, ...]
    x_half1 = x[..., 0::2]
    x_half2 = x[..., 1::2]
    x_rotated = torch.stack((-x_half2, x_half1), dim=-1).flatten(-2)
    
    return (x * cos) + (x_rotated * sin)

Because RoPE is applied directly to $q_m$ and $k_n$ before entering the scaled dot-product attention kernel, it introduces zero memory overhead during the attention computation itself. This makes RoPE compatible with memory-efficient attention algorithms like FlashAttention (Dao et al., 2022).

Long-Range Attention Decay

An essential property of RoPE is the natural decay of the attention score as the relative distance $|m - n|$ grows.

The expected inner product over random query and key distributions decomposes into a summation of Fourier modes:

$$\mathbb{E}[\langle f_q(x_m, m), f_k(x_n, n) \rangle] \propto \sum_{i=1}^{d/2} \cos((m - n)\theta_i)$$

As relative distance $m - n$ increases, the varying frequencies $\theta_i$ oscillate at different rates, causing destructive interference. By the Riemann-Lebesgue lemma, the sum over non-zero frequencies decays with increasing $|m - n|$.

This mathematical property introduces an inductive bias into transformer attention: tokens in close proximity have higher baseline affinity, while distant tokens require stronger semantic correlation to overcome phase cancellation.

Scaling RoPE to Long Context Windows

While RoPE theoretically supports arbitrary relative positions, models pre-trained on sequence length $L_{\text{train}}$ fail when evaluated on lengths $L > L_{\text{train}}$.

At positions beyond $L_{\text{train}}$, low-frequency channels experience unseen rotational phases, while high-frequency channels rotate into out-of-distribution angular domains. Several techniques have been developed to extend context windows without re-training from scratch.

Position Interpolation (PI)

Introduced by Chen et al. in Extending Context Window of Large Language Models via Position Interpolation (2023), Position Interpolation downscales position indices linearly by a factor $s = L_{\text{new}} / L_{\text{train}}$:

$$m' = \frac{m}{s}$$

This maps the extended range $[0, L_{\text{new}}]$ back into the pre-trained phase range $[0, L_{\text{train}}]$.

While Position Interpolation prevents out-of-distribution phases and stabilizes fine-tuning within 1,000 steps, it uniformly compresses all frequencies. High frequencies (which capture local syntactic and token-order relationships) are compressed, reducing the model's ability to discriminate adjacent tokens.

NTK-Aware RoPE Scaling

To address frequency compression, the open-source community (initiated by user bloc97) developed Neural Tangent Kernel (NTK)-Aware scaling.

Rather than scaling positions linearly, NTK-Aware scaling modifies the base frequency $\theta_{\text{base}}$:

$$\theta_{\text{base}}' = \theta_{\text{base}} \cdot s^{\frac{d}{d-2}}$$

By scaling the base frequency:

• High-frequency dimensions (low $i$) undergo minimal change, preserving fine-grained local positional resolution.
• Low-frequency dimensions (high $i$) are stretched substantially, enabling long-range position discrimination without phase explosion.

YaRN (Yet another RoPE extensioN)

Peng et al. introduced YaRN: Efficient Context Window Extension of Large Language Models (2023) to unify interpolation and extrapolation across three distinct frequency regimes:

1. High Frequencies (Wavelength $\lambda < L_{\text{train}}$): No interpolation ($s=1$). High frequencies complete multiple full rotations within the pre-trained window; interpolating them degrades local resolution.
2. Low Frequencies (Wavelength $\lambda > L_{\text{train}}$): Full interpolation with factor $s$. These dimensions have not completed a full rotation during pre-training and require scaling to prevent extrapolation into unseen angles.
3. Medium Frequencies: A smooth linear ramp function $\gamma(r)$ transitions between extrapolation and interpolation.

YaRN also incorporates an attention temperature scaling factor $\sqrt{t}$ where $t = 0.1 \ln(s) + 1$. Because long sequences disperse attention weights across more tokens (increasing attention entropy), temperature scaling sharpens the softmax distribution back to pre-training levels.

Architectural Trade-Offs

Comparing positional embedding strategies highlights key trade-offs in attention design:

• Absolute Sinusoidal: Added directly to input embeddings. No kernel overhead, but lacks relative distance invariance and exhibits poor long-context extrapolation.
• Learnable Absolute: Fixed vocabulary of learned position vectors. No kernel overhead, but bounded strictly to maximum trained sequence length with zero extrapolation capacity.
• Relative Bias (Shaw / T5): Direct $O(L^2)$ matrix bias added to attention logits. Captures relative distances accurately, but incurs high memory and kernel overhead, limiting compatibility with fused attention implementations like FlashAttention.
• Rotary Position Embedding (RoPE): Orthogonal in-place rotation of query and key projections. Achieves exact relative distance invariance, introduces zero memory overhead during attention computation, provides natural Fourier long-range decay, and supports context extension via Position Interpolation, NTK-aware scaling, and YaRN.

Sources

• Su, J., Lu, Y., Pan, S., Wen, B., & Liu, Y. (2021). RoFormer: Enhanced Transformer with Rotary Position Embedding. arXiv preprint arXiv:2104.09864.
• Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention Is All You Need. Advances in Neural Information Processing Systems (NeurIPS 2017).
• Shaw, P., Uszkoreit, J., & Vaswani, A. (2018). Self-Attention with Relative Position Representations. Proceedings of NAACL-HLT 2018.
• Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., & Salakhutdinov, R. (2019). Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. Proceedings of ACL 2019.
• Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., & Liu, P. J. (2020). Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research (JMLR).
• Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. Advances in Neural Information Processing Systems (NeurIPS 2022).
• Chen, S., Wong, S., Chen, L., & Tian, Y. (2023). Extending Context Window of Large Language Models via Position Interpolation. arXiv preprint arXiv:2306.15595.
• Peng, B., Quesnelle, J., Fan, H., & Shippole, E. (2023). YaRN: Efficient Context Window Extension of Large Language Models. International Conference on Learning Representations (ICLR 2024).