Self-Distillation with No Labels (DINO): How Momentum Teachers, Centering, and Sharpening Emerge Semantic Attention in Vision Transformers

When the Vision Transformer (ViT) was introduced by Dosovitskiy et al. in 2020, standard wisdom suggested that transformers required massive supervised corpora (such as JFT-300M) to overcome their lack of convolutional inductive biases. Unlike Convolutional Neural Networks (CNNs), which bake translation equivariance and local receptive fields directly into their kernel structures, standard ViTs possess global receptive fields from layer one. When trained with conventional supervised cross-entropy on ImageNet-1K, ViTs often learned scattered, non-localized attention patterns, and their internal features failed to segment visual objects cleanly without dense pixel-level supervision.

In 2021, Mathilde Caron and researchers at Facebook AI Research published Emerging Properties in Self-Supervised Vision Transformers (DINO). DINO (short for self-DIstillation with NO labels) demonstrated that self-supervised pre-training fundamentally unlocks the representational capacity of Vision Transformers. Without a single human annotation, class label, or segmentation mask, DINO-trained ViTs automatically learn scene layouts, isolate object boundaries within their self-attention maps, and produce feature embeddings where zero-shot $k$-nearest neighbor ($k$-NN) classifiers rival supervised linear probes.

The Self-Distillation Architecture

Self-distillation frames self-supervised representation learning as a student-teacher knowledge transfer problem where both models share identical neural network architectures. Rather than training against a pre-existing, static teacher, the student and teacher evolve concurrently during training.

       Image Input (X)
         /         \
   Global Crop    Global + Local Crops
       |                   |
  +----+-----+        +----+-----+
  | Teacher  |        | Student  |
  | Backbone |        | Backbone |
  | (ViT/EMA)|        |  (ViT)   |
  +----+-----+        +----+-----+
       |                   |
  +----+-----+        +----+-----+
  | Projection|       | Projection|
  |  Head g_t |       |  Head g_s |
  +----+-----+        +----+-----+
       |                   |
   Centering (c)           |
   & Sharpening            |
     (tau_t)            (tau_s)
       \                  /
        \                /
     Cross-Entropy Loss H(P_t, P_s)
                 |
        Backward Gradient (Student Only)
                 |
      EMA Update: theta_t <-- lambda * theta_t + (1 - lambda) * theta_s

1. Dual-Network Formulation

The system consists of two networks:

• Student Network $g_{\theta_s}$: Parameterized by weights $\theta_s$, trained directly via gradient descent.
• Teacher Network $g_{\theta_t}$: Parameterized by weights $\theta_t$, updated exclusively through an Exponential Moving Average (EMA) of the student weights.

Both networks consist of a Vision Transformer backbone (mapping an image $x$ to a class token embedding $f(x)$) followed by a 3-layer Multi-Layer Perceptron (MLP) projection head. The projection head features hidden dimensions of 2048, a bottleneck dimension of 256, $L_2$ weight normalization, and a $K$-dimensional normalized prototype layer (typically $K = 65,536$). The output logits represent unnormalized scores over $K$ virtual prototype dimensions.

2. The Momentum Teacher Update

The teacher parameters $\theta_t$ do not receive gradients ($\frac{\partial \mathcal{L}}{\partial \theta_t} = 0$). Instead, after every optimization step of the student, the teacher is updated using a Polyak-Ruppert moving average:

$$\theta_t \leftarrow \lambda \theta_t + (1 - \lambda) \theta_s$$

The momentum parameter $\lambda$ follows a cosine schedule during training, ramping smoothly from an initial value of $\lambda_0 = 0.996$ to $\lambda_1 = 1.0$.

This momentum update causes the teacher to behave as a temporal ensemble of past student states, analogous to Mean Teacher frameworks and Bootstrap Your Own Latent (BYOL). Because the teacher integrates weights over thousands of optimization steps, its output distributions remain smooth and stable, providing consistent pseudo-targets that guide the student.

Collapse Dynamics: Centering vs. Sharpening

Self-supervised learning without negative samples faces two fundamental representation collapse modes:

1. Uniform Collapse: Output probabilities flatten across all $K$ dimensions ($P(x)^{(k)} \to 1/K$ for all $k$), destroying all discriminative capacity.
2. Delta (One-Hot) Collapse: A single output dimension dominates across the entire dataset ($P(x)^{(c)} \to 1$ for some fixed index $c$, regardless of input $x$).

Previous frameworks prevented collapse using contrastive negative pairs (such as SimCLR and MoCo), batch normalization layers (BYOL), or stop-gradient architectures with asymmetric predictors (SimSiam).

DINO avoids negative samples, memory banks, and batch normalization entirely. Instead, it eliminates both collapse modes through the mathematical interplay of two operations: Centering and Sharpening.

The Centering Operator

Centering adds a dynamic bias vector $c \in \mathbb{R}^K$ to the teacher's raw logits $g_{\theta_t}(x)$, subtracting the mean activation:

$$g_t(x) = g_{\theta_t}(x) - c$$

The center vector $c$ is tracked as an exponential moving average of batch representations across training:

$$c \leftarrow m c + (1 - m) \frac{1}{B} \sum_{i=1}^B g_{\theta_t}(x_i)$$

where $B$ is the batch size and $m$ is the center momentum coefficient (typically $m = 0.9$).

Mechanics: If one prototype dimension begins to fire frequently across different images, its corresponding component in $c$ increases, suppressing its logit value in subsequent steps. Centering prevents any single dimension from dominating the output, ruling out delta collapse. However, unconstrained centering pulls all logits toward equality, encouraging uniform collapse.

The Sharpening Operator

Sharpening normalizes the centered logits into a probability distribution over the $K$ prototypes using a temperature-scaled softmax:

$$P_t(x)^{(i)} = \frac{\exp\left((g_{\theta_t}(x)^{(i)} - c^{(i)}) / \tau_t\right)}{\sum_{k=1}^K \exp\left((g_{\theta_t}(x)^{(k)} - c^{(k)}) / \tau_t\right)}$$

$$P_s(x)^{(i)} = \frac{\exp\left(g_{\theta_s}(x)^{(i)} / \tau_s\right)}{\sum_{k=1}^K \exp\left(g_{\theta_s}(x)^{(k)} / \tau_s\right)}$$

where $\tau_t$ and $\tau_s$ govern distribution entropy.

Crucially, DINO enforces asymmetric temperatures: the teacher temperature $\tau_t$ is strictly lower than the student temperature $\tau_s$. The student temperature is fixed at $\tau_s = 0.1$, while the teacher temperature $\tau_t$ warms up linearly from $0.04$ to $0.07$ over the first 30 epochs.

Mechanics: A low teacher temperature $\tau_t$ sharpens the output distribution, penalizing uniform entropy and forcing the teacher to make high-confidence predictions. While sharpening alone induces delta collapse, combining centering with sharpening forces the distribution to remain peaked while ensuring that peaks are uniformly distributed across all $K$ prototypes over the dataset.

| Regularization Component | Mathematical Formulation | Primary Failure Mode Prevented | Secondary Risk Induced | | :--- | :--- | :--- | :--- | | Centering ($c$) | $g_t(x) \leftarrow g_{\theta_t}(x) - c$ | Delta / One-Hot Collapse | Uniform Entropy Collapse | | Sharpening ($\tau_t$) | $\text{Softmax}(g_t / \tau_t), \quad \tau_t < \tau_s$ | Uniform Entropy Collapse | Delta / One-Hot Collapse | | Combined Equilibrium | $\text{Softmax}((g_{\theta_t}(x) - c) / \tau_t)$ | Both Modes Eliminated | None (Stable Training) |

Multi-Crop Training: Local-to-Global View Invariance

To force the student network to learn part-to-whole hierarchies, DINO adopts a multi-crop data augmentation strategy (Caron et al., 2020):

1. Global Views ($x_1^g, x_2^g$): Two standard crops at $224 \times 224$ resolution, each covering a large area ($>50\%$) of the source image.
2. Local Views ($V_L$): Several small crops (typically $V = 6$ or $8$) at $96 \times 96$ resolution, covering small sub-regions ($<50\%$) of the source image.

+-------------------------------------------------------------+
| Source Image (Full Scene)                                   |
|   +--------------------------+                              |
|   | Global Crop 1 (>50%)     |      +------------------+    |
|   | (Passed to Teacher &     |      | Local Crop 1     |    |
|   |  Student, 224x224)       |      | (<50%, 96x96)    |    |
|   +--------------------------+      | (Student Only)   |    |
|                                     +------------------+    |
|            +--------------------------+                     |
|            | Global Crop 2 (>50%)     |   +---------------+ |
|            | (Passed to Teacher &     |   | Local Crop 2  | |
|            |  Student, 224x224)       |   | (Student Only)| |
|            +--------------------------+   +---------------+ |
+-------------------------------------------------------------+

The Asymmetric Information Bottleneck

The teacher processes only the 2 global crops, ensuring that its representations always reflect global context and high-level scene composition.

The student processes all crops ($2 \text{ global} + V \text{ local}$).

The optimization objective minimizes the cross-entropy loss between student predictions and teacher targets across all asymmetric view pairs:

$$\mathcal{L}_{\text{DINO}} = \sum_{x \in \{x_1^g, x_2^g\}} \sum_{\substack{x' \in V_{\text{all}} \\ x' \neq x}} H\left(P_t(x), P_s(x')\right)$$

where $H(a, b) = -\sum_{k=1}^K a^{(k)} \log b^{(k)}$.

Because the student is presented with a local patch (for instance, the paw of a dog) and tasked with predicting the global representation generated by the teacher (which saw the entire dog), the student is compelled to infer global context from local parts. This local-to-global objective prevents the model from relying on superficial background texture shortcuts.

Emergent Semantic Attention and Evaluation

The most striking discovery in the DINO paper is the spontaneous emergence of clean semantic segmentation inside the self-attention maps of the Vision Transformer.

1. Unsupervised Semantic Segmentation

In a standard Vision Transformer, the [CLS] token interacts with all patch tokens through multi-head self-attention:

$$\text{Attention}(Q, K, V) = \text{Softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

By extracting the attention weights of the [CLS] token query with respect to all spatial patch keys in the final transformer layer, one obtains a 2D spatial heatmap over the input image.

In supervised ViTs, these attention maps often disperse across high-frequency edges and background artifacts. In DINO-trained ViTs, the [CLS] attention maps automatically mask out background noise and tightly segment foreground objects. Furthermore, different attention heads in the final layer spontaneously specialize in distinct semantic entities (such as separating individual riders from bicycles or isolating distinct anatomical components of an animal).

Self-Attention Heatmap Decomposition:
Input Image ----> [ ViT Backbone ] ----> Final Layer [CLS] Attention
                                         |-- Head 1: Foreground Object Boundary
                                         |-- Head 2: Fine Detail / Sub-Part
                                         |-- Head 3: Context / Secondary Object

2. Quantitative Benchmarks

DINO demonstrated that self-supervised ViT features are directly usable without linear probe fine-tuning, evaluated via zero-shot $k$-nearest neighbors ($k$-NN) on frozen representations:

| Architecture | Patch Size | Parameters | ImageNet Top-1 ($k$-NN) | ImageNet Top-1 (Linear Probe) | | :--- | :--- | :--- | :--- | :--- | | ResNet-50 | N/A | 23M | 67.5% | 75.3% | | ViT-S | 16 | 21M | 74.5% | 77.0% | | ViT-S | 8 | 21M | 79.7% | 80.1% | | ViT-B | 16 | 85M | 76.1% | 78.2% | | ViT-B | 8 | 85M | 78.3% | 80.1% |

With smaller patch sizes (such as $8 \times 8$ instead of $16 \times 16$), the spatial resolution of the visual tokens increases $4\times$, boosting both linear probe and zero-shot $k$-NN accuracy to $80.1\%$ and $79.7\%$ on ImageNet-1K, surpassing supervised ResNet-50 baselines.

Architectural Evolution: DINOv2

In 2023, Meta AI expanded the framework with DINOv2 (Oquab et al., 2023), scaling self-supervised vision representations to foundation model capacity. DINOv2 introduced three primary architectural refinements:

1. Joint Patch-Level and Image-Level Objectives: Integrated a patch-level masked image modeling loss (iBOT, Zhou et al., 2021), training the student to predict both the global [CLS] token and masked spatial patch tokens simultaneously.
2. Untied Weight Decay and KoLeo Regularizer: Added the Kozachenko-Leonenko differential entropy estimator to maximize the uniformity of feature embeddings across the unit hypersphere.
3. Data Curation and Scaled Hardware: Trained on the curated LVD-142M dataset using FlashAttention, FP16 mixed precision, and model sizes up to ViT-g/14 (1.1 billion parameters).

DINOv2 features serve as standard visual backbones across dense prediction tasks, including monocular depth estimation, surface normal estimation, visual correspondence, and multimodal LLM vision encoders.

Sources

• Emerging Properties in Self-Supervised Vision Transformers (arXiv:2104.14294) - Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, Armand Joulin (2021).
• DINOv2: Learning Robust Visual Features without Supervision (arXiv:2304.07193) - Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Jaewoo Kang, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Hervé Jégou (2023).
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (arXiv:2010.11929) - Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, Neil Houlsby (2020).
• Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (arXiv:2006.07733) - Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, Michal Valko (2020).
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments (arXiv:2006.09882) - Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, Armand Joulin (2020).
• iBOT: Image BERT Pre-Training with Online Tokenizer (arXiv:2111.07832) - Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, Tao Kong (2021).