Summary · Chapter 7

Self-supervised features: DINO to DINOv3

ModelIdea
DINO (2021)Self-distillation without labels; the teacher is a momentum encoder; multi-crop; abstract reports 78.3% k-NN (small ViT) and 80.1% linear (ViT-Base) on ImageNet
DINOv2 (2023)Curated data, a 1-billion-parameter ViT distilled into smaller models
DINOv3 (2025)More scale; Gram anchoring against dense-feature degradation in long training; distilled small models (your ViT-S/16)
Registers (2023)Extra tokens absorb the high-norm artifact tokens; this teacher has four storage tokens

Linear probe: a single linear layer on frozen features; accuracy shows what is linearly readable. Recorded (800 holdout images, one seed): Gaussian model on 512 px target 0.889; teacher CLS 0.921; teacher patch mean 0.866.

Why frozen: fixed target, caches computed once, semantics inherited. Cost: the teacher is needed at inference (roadmap P4).

Defence answer: “The semantics come from a frozen, self-supervised DINOv3-S/16; my contribution is a compact, amortised way to place Gaussians and fit features to it, not the semantics.”

Do not quote a Gram-anchoring mechanism or numbers from the abstract; it gives neither.