Part II · The teacher signal · Chapter 7

Self-supervised features: DINO to DINOv3

About 18 minutes. Chapter 6 showed how a vision transformer makes a grid of vectors. This chapter explains why those vectors mean anything, and why your teacher is DINOv3.

7.1 Why this matters for your claim

Your abstract says the Gaussians are “semantically relevant”. The only source of semantics in the system is the teacher. A reviewer will ask where its features come from, whether they are trustworthy on a dense grid, and why they are frozen. This chapter gives you those answers.

By the end you can explain in plain words how DINO trains a network without labels, what DINOv2 and DINOv3 added, what register tokens and Gram anchoring fix, how a linear probe reads features, and why the teacher stays frozen in this project.

First, write what you think “self-supervised” means and why labels might not be needed.

7.2 Learning without labels

A supervised network learns from pairs (image, label made by a human). A self-supervised network is given only images, and a task invented so that the answer is already in the data. A language model hides the next word. An image model can hide pixels, as in masked autoencoders[He et al., 2021], or ask that two views of one picture get the same description.

Self-supervised learning. Training on a task built from the data itself, with no human labels. The aim is a network whose internal features are useful for many later tasks.

The question for your work is not “does it classify well” but “do the patch vectors, one per square of the image, describe what is in that square in a way that is stable and shareable”? That is a dense-feature question, and it is why this chapter ends on DINOv3.

7.3 DINO: a network teaches itself

DINO[Caron et al., 2021] describes its method as self-distillation with no labels. The idea has three parts, stated at the level the paper's abstract supports:

  1. Show the network several views of the same image, made by cropping and rescaling (the abstract names multi-crop training as important).
  2. One copy of the network, the student, must produce the same output as another copy, the teacher, on a different view of the same picture.
  3. The teacher is a momentum encoder. It is not trained by gradients but follows the student as a slowly moving average of its weights.
Self-distillation. Distillation is training one network to copy another's outputs. In self-distillation the “other network” is a copy of the learner itself, so no outside teacher or labels are needed.

The paper reports that the resulting ViT features contain explicit information about the semantic segmentation of an image, more clearly than in supervised ViTs or convolutional networks. It also reports 78.3% top-1 on ImageNet with a k-nearest-neighbour classifier on a small ViT, and 80.1% with a linear evaluation on ViT-Base (numbers from the abstract page). The abstract also credits small patches as important.

Two meanings of “teacher” are now in play. Inside DINO's training, the teacher is the momentum copy. In your project, the teacher is the finished, frozen DINOv3 network whose patch features you imitate. Keep them apart in conversation.

7.4 DINOv2 and DINOv3

DINOv2[Oquab et al., 2023] argues that general-purpose features come from the same kind of training, provided the data is curated and diverse. Its authors build an automated pipeline to assemble that dataset, train a 1-billion-parameter ViT and distil it into smaller models. They report that these beat OpenCLIP, the strongest general-purpose feature extractor available at the time, on most image-level and pixel-level benchmarks.

DINOv3[Siméoni et al., 2025] scales the data and the model up further. It adds a method called Gram anchoring and some later steps that improve flexibility across resolution and model size. The family includes distilled small models: your teacher, ViT-S/16, is one of them. The abstract says the model produces high-quality dense features and beats specialised methods without fine-tuning.

ModelYearMain addition
DINO2021Label-free self-distillation with a momentum teacher
DINOv22023Curated data, scale, distillation into small models
DINOv32025Larger scale, Gram anchoring, flexibility steps

7.5 Two problems in dense features

The patch vectors are what your project uses, and they have two known flaws.

Artifacts: the register fix

Darcet and colleagues found that ViTs, supervised and self-supervised, produce a few high-norm tokens, mostly in low-information background regions. The network seems to reuse those patches as scratch space for its own computation[Darcet et al., 2023]. The patch no longer describes its own pixels. Their fix is to add extra tokens to the input sequence, called registers, so the network has somewhere else to compute. The abstract reports that this removes the problem, and gives smoother feature and attention maps.

Register token. An extra learned token in the input sequence that carries no image content. It is discarded at the output and exists to absorb global computation.

Your teacher has four of these. The runtime code calls them storage tokens and lists them in the sequence [CLS, 4 storage, G*G patches]. Chapter 6 showed that the output step drops them.

Degradation over long training: Gram anchoring

The DINOv3 abstract says dense feature maps degrade during long training schedules, a known but unsolved problem, and that Gram anchoring is the new method aimed at it. The abstract page gives no mechanism or numbers, so do not quote any. A Gram matrix of a set of vectors is the table of all pairwise dot products. The name suggests the method constrains how similar every patch is to every other, but the abstract does not say how. Read the paper's method section before you describe the mechanism in detail.

The roadmap's loss mentions a “Gram” term in your own training. That is a separate use of the idea inside your student. DINOv3's Gram anchoring happens while the teacher is trained, and you never touch it.

7.6 Reading features with a linear probe, and why freeze

How do you know what a feature vector “knows”? Freeze the features and train only one linear layer on top to predict a label.

Linear probe. A single linear classifier trained on frozen features. High accuracy means the label is already linearly readable from the features, so the network encodes it explicitly.

Your project has a probe of this kind on the Gaussian features. Recorded values on the 800 holdout images, one seed: the 512 px Gaussian model reaches top-1 0.889, against the teacher's own CLS vector at 0.921 and its mean patch vector at 0.866 (roadmap.md run log, 14 November). On the 16×16 target the Gaussian models reach 0.869 to 0.871. The probe was insensitive to where the Gaussians sit, which Chapter 19 returns to.

Why the teacher is frozen.

  • A fixed target. A moving teacher would make “error” change meaning during training.
  • Inherited semantics. You do not train anything to be semantic. The teacher already is, which also means your features are copies of its features, as the roadmap's known weaknesses say.
  • Cost. The caches are computed once. Training then never runs the teacher.

The price: at inference the teacher must still run, to get the features to fit. The roadmap lists removing that dependence as publication track P4. Present it as a limitation, not a hidden one.

7.7 Check yourself

1. In DINO, what is the “teacher” network during training?

DINO is self-distillation without labels. The teacher is a momentum encoder that follows the student's weights.

2. What problem do register tokens address?

Registers give the network scratch tokens so it stops hijacking background patches. Coarse grids are Chapter 8, and Gram anchoring targets long-schedule degradation.

3. What does a high linear-probe accuracy show?

A probe tests whether information is explicitly present. A self-supervised network never saw the labels, and probe accuracy says nothing about grid resolution.

4. A reviewer says your features are “just copied”. What is your answer?

The teacher is frozen and the features are fitted to it. Honesty here is stronger than overclaiming; the contribution is the compact placement and fitting, not the semantics.

5. Why does freezing the teacher help your experiments?

A fixed target keeps dense MSE comparable across runs. The teacher is still needed at inference, which is the P4 weakness.

7.8 Close the loop

Without scrolling up, explain where your teacher's features come from, what its two known flaws are, and why it is frozen. Then compare with your first answer.

A model answer

“DINOv3 is a self-supervised ViT trained without labels, in the DINO line: a student matches a slowly moving teacher copy on different views of the same image. DINOv2 added curated data and scale, and DINOv3 added more scale and Gram anchoring to stop dense features degrading in long training. Register tokens fix high-norm artifact patches, and my teacher carries four of them. I freeze the teacher so the target is fixed and the caches are computed once. The consequence is that the semantics are inherited and the teacher is still needed at inference. Linear probes show how much label information the features carry.”

Read next Siméoni et al., “DINOv3” (2025), the sections on Gram anchoring and dense evaluation. It is the paper that produced your teacher and the one a reviewer will expect you to know.

Something unclear, or want to rehearse how you would say this to a reviewer? Ask your teacher in the chat.

One-page summary of this chapter ·