Self-supervised features: DINO to DINOv3
| Model | Idea |
|---|---|
| DINO (2021) | Self-distillation without labels; the teacher is a momentum encoder; multi-crop; abstract reports 78.3% k-NN (small ViT) and 80.1% linear (ViT-Base) on ImageNet |
| DINOv2 (2023) | Curated data, a 1-billion-parameter ViT distilled into smaller models |
| DINOv3 (2025) | More scale; Gram anchoring against dense-feature degradation in long training; distilled small models (your ViT-S/16) |
| Registers (2023) | Extra tokens absorb the high-norm artifact tokens; this teacher has four storage tokens |
Linear probe: a single linear layer on frozen features; accuracy shows what is linearly readable. Recorded (800 holdout images, one seed): Gaussian model on 512 px target 0.889; teacher CLS 0.921; teacher patch mean 0.866.
Why frozen: fixed target, caches computed once, semantics inherited. Cost: the teacher is needed at inference (roadmap P4).
Defence answer: “The semantics come from a frozen, self-supervised DINOv3-S/16; my contribution is a compact, amortised way to place Gaussians and fit features to it, not the semantics.”
Do not quote a Gram-anchoring mechanism or numbers from the abstract; it gives neither.