7.1 Why this matters for your claim
Your abstract says the Gaussians are “semantically relevant”. The only source of semantics in the system is the teacher. A reviewer will ask where its features come from, whether they are trustworthy on a dense grid, and why they are frozen. This chapter gives you those answers.
First, write what you think “self-supervised” means and why labels might not be needed.
7.2 Learning without labels
A supervised network learns from pairs (image, label made by a human). A self-supervised network is given only images, and a task invented so that the answer is already in the data. A language model hides the next word. An image model can hide pixels, as in masked autoencoders[He et al., 2021], or ask that two views of one picture get the same description.
The question for your work is not “does it classify well” but “do the patch vectors, one per square of the image, describe what is in that square in a way that is stable and shareable”? That is a dense-feature question, and it is why this chapter ends on DINOv3.
7.3 DINO: a network teaches itself
DINO[Caron et al., 2021] describes its method as self-distillation with no labels. The idea has three parts, stated at the level the paper's abstract supports:
- Show the network several views of the same image, made by cropping and rescaling (the abstract names multi-crop training as important).
- One copy of the network, the student, must produce the same output as another copy, the teacher, on a different view of the same picture.
- The teacher is a momentum encoder. It is not trained by gradients but follows the student as a slowly moving average of its weights.
The paper reports that the resulting ViT features contain explicit information about the semantic segmentation of an image, more clearly than in supervised ViTs or convolutional networks. It also reports 78.3% top-1 on ImageNet with a k-nearest-neighbour classifier on a small ViT, and 80.1% with a linear evaluation on ViT-Base (numbers from the abstract page). The abstract also credits small patches as important.
Two meanings of “teacher” are now in play. Inside DINO's training, the teacher is the momentum copy. In your project, the teacher is the finished, frozen DINOv3 network whose patch features you imitate. Keep them apart in conversation.
7.4 DINOv2 and DINOv3
DINOv2[Oquab et al., 2023] argues that general-purpose features come from the same kind of training, provided the data is curated and diverse. Its authors build an automated pipeline to assemble that dataset, train a 1-billion-parameter ViT and distil it into smaller models. They report that these beat OpenCLIP, the strongest general-purpose feature extractor available at the time, on most image-level and pixel-level benchmarks.
DINOv3[Siméoni et al., 2025] scales the data and the model up further. It adds a method called Gram anchoring and some later steps that improve flexibility across resolution and model size. The family includes distilled small models: your teacher, ViT-S/16, is one of them. The abstract says the model produces high-quality dense features and beats specialised methods without fine-tuning.
| Model | Year | Main addition |
|---|---|---|
| DINO | 2021 | Label-free self-distillation with a momentum teacher |
| DINOv2 | 2023 | Curated data, scale, distillation into small models |
| DINOv3 | 2025 | Larger scale, Gram anchoring, flexibility steps |
7.5 Two problems in dense features
The patch vectors are what your project uses, and they have two known flaws.
Artifacts: the register fix
Darcet and colleagues found that ViTs, supervised and self-supervised, produce a few high-norm tokens, mostly in low-information background regions. The network seems to reuse those patches as scratch space for its own computation[Darcet et al., 2023]. The patch no longer describes its own pixels. Their fix is to add extra tokens to the input sequence, called registers, so the network has somewhere else to compute. The abstract reports that this removes the problem, and gives smoother feature and attention maps.
Your teacher has four of these. The runtime code calls them storage tokens and lists them in the sequence [CLS, 4 storage, G*G patches]. Chapter 6 showed that the output step drops them.
Degradation over long training: Gram anchoring
The DINOv3 abstract says dense feature maps degrade during long training schedules, a known but unsolved problem, and that Gram anchoring is the new method aimed at it. The abstract page gives no mechanism or numbers, so do not quote any. A Gram matrix of a set of vectors is the table of all pairwise dot products. The name suggests the method constrains how similar every patch is to every other, but the abstract does not say how. Read the paper's method section before you describe the mechanism in detail.
The roadmap's loss mentions a “Gram” term in your own training. That is a separate use of the idea inside your student. DINOv3's Gram anchoring happens while the teacher is trained, and you never touch it.
7.6 Reading features with a linear probe, and why freeze
How do you know what a feature vector “knows”? Freeze the features and train only one linear layer on top to predict a label.
Your project has a probe of this kind on the Gaussian features. Recorded values on the 800 holdout images, one seed: the 512 px Gaussian model reaches top-1 0.889, against the teacher's own CLS vector at 0.921 and its mean patch vector at 0.866 (roadmap.md run log, 14 November). On the 16×16 target the Gaussian models reach 0.869 to 0.871. The probe was insensitive to where the Gaussians sit, which Chapter 19 returns to.
Why the teacher is frozen.
- A fixed target. A moving teacher would make “error” change meaning during training.
- Inherited semantics. You do not train anything to be semantic. The teacher already is, which also means your features are copies of its features, as the roadmap's known weaknesses say.
- Cost. The caches are computed once. Training then never runs the teacher.
The price: at inference the teacher must still run, to get the features to fit. The roadmap lists removing that dependence as publication track P4. Present it as a limitation, not a hidden one.
7.7 Check yourself
1. In DINO, what is the “teacher” network during training?
2. What problem do register tokens address?
3. What does a high linear-probe accuracy show?
4. A reviewer says your features are “just copied”. What is your answer?
5. Why does freezing the teacher help your experiments?
7.8 Close the loop
Without scrolling up, explain where your teacher's features come from, what its two known flaws are, and why it is frozen. Then compare with your first answer.
A model answer
“DINOv3 is a self-supervised ViT trained without labels, in the DINO line: a student matches a slowly moving teacher copy on different views of the same image. DINOv2 added curated data and scale, and DINOv3 added more scale and Gram anchoring to stop dense features degrading in long training. Register tokens fix high-norm artifact patches, and my teacher carries four of them. I freeze the teacher so the target is fixed and the caches are computed once. The consequence is that the semantics are inherited and the teacher is still needed at inference. Linear probes show how much label information the features carry.”
Something unclear, or want to rehearse how you would say this to a reviewer? Ask your teacher in the chat.