6.1 Why the teacher matters
Everything your Gaussians store is copied from a teacher: a frozen network that reads the image and writes a 384-number vector for every patch. The error you report is the distance between your field and the teacher's field. If you cannot say what the teacher's field is, you cannot say what your number measures.
First, write down what you think happens between “image goes in” and “feature map comes out”. A guess is fine.
6.2 Cut the image into patches
A vision transformer, or ViT, does not look at pixels one at a time. It cuts the image into small squares and treats each square as one item, the way a language model treats a word. The original paper made this the title: an image is worth 16×16 words[Dosovitskiy et al., 2020]. It showed that a pure Transformer applied to sequences of image patches does well on image classification.
| Input side S | Patches per side G | Patches G² |
|---|---|---|
| 256 px | 16 | 256 |
| 512 px | 32 | 1024 |
Your teacher is the DINOv3 ViT-S with patch size 16. Doubling the side of the image multiplies the number of patches by four. Look at what that means for the grid:
Both panels show the same picture. Each small square is one token. On the right the squares are half as wide on the picture, because the network still reads 16 px squares but the image has been enlarged to twice the side. Chapter 8 returns to what the left grid hides.
6.3 From a square of pixels to a token
A 16×16 patch of colour pixels holds 16 × 16 × 3 = 768 numbers. A learned linear map turns those 768 numbers into one vector of the network's working width. For the ViT-S that width is 384, the same D = 384 you met in Chapter 1.
In this project frozen_dino_runtime_grid.py shows the first step plainly: a pack kernel rearranges the image into G² rows of 768 normalised pixel values, one row per patch, ready for that linear map. The pixel values are first shifted and scaled by the standard ImageNet colour statistics.
Nothing so far says where a patch was. A bag of 256 vectors has no order. The next step fixes that.
6.4 Position and the class token
Attention (next step) treats its input as an unordered set. To let the network know that one patch sits above another, the model injects position information. Classic ViTs add a learned vector for each grid position to the token. Your teacher uses a related scheme called rotary position embedding: position is mixed into the attention calculation itself. The code comment says the coordinates are normalised per axis as (i + 0.5) / G × 2 − 1, which is the same [−1, 1] frame your Gaussians use.
Two kinds of extra tokens travel with the patch tokens:
- A class token (CLS). One extra vector that belongs to no patch. Through attention it gathers information from the whole image and is read out at the end as a summary of the picture. Your cache stores it as the global feature.
- A few storage tokens, four in this teacher. Chapter 7 explains why they exist.
The teacher's token sequence for a 256 px image is therefore 1 + 4 + 256 = 261 tokens, and for a 512 px image 1 + 4 + 1024 = 1029. The code comment in frozen_dino_runtime_grid.py gives the 261.
6.5 Attention in one paragraph
A transformer is a stack of identical layers. Inside each layer every token produces three vectors of its own: a query (what it is looking for), a key (what it offers) and a value (what it would hand over). A token compares its query with every token's key by dot product, turns the scores into weights that sum to one with a softmax, and takes the weighted average of the values. The result is added back to the token. Many such averages run in parallel (called heads), and a small feed-forward network follows. After many layers each patch token has mixed in information from every other patch, so the vector for a patch describes the patch in the context of the whole image.
Two consequences matter for your work.
- Cost. Every token compares with every other, so the number of pairs is the token count squared. At 256 tokens that is 65,536 pairs; at 1024 tokens it is 1,048,576, sixteen times as many. Four times the tokens costs sixteen times the attention pairs.
- Features are contextual. A patch of sky and a patch of sky next to a bird do not get identical vectors. The teacher field is not a local filter of the image.
6.6 The output is a grid of vectors
After the last layer the code does a final normalisation and then extracts the patch tokens and the CLS token. Discard the storage tokens, reshape the G² patch tokens back to G × G, and you have the teacher field.
Chapter 4 fitted Gaussian features to exactly this object: the target matrix T has one row per grid cell. Two small details in this project's pipeline:
- The image is resized square with bicubic interpolation, with no crop, so the whole picture is always in view (
cache_frozen_patches_fine_20261113.py). - Patch tokens are stored as 16-bit numbers in the cache, and the models normalise each token to unit length themselves. The cache does not.
The grid is a result, not a choice you made. The teacher decides the sampling pattern: one vector per 16×16 pixels. To get a finer grid you must change the input, not the Gaussians. That is the subject of Chapter 8.
6.7 Check yourself
1. A 512 px image goes through a patch-size-16 ViT. How many patch tokens result?
2. Why does the network need position information at all?
3. Why is the 16×16 feature map an explicit representation?
4. Why is a patch token more than a description of its own square?
5. Going from 256 px to 512 px multiplies the attention pair count by what?
6.8 Close the loop
Without scrolling up, explain how a 256 px image becomes the teacher field your Gaussians fit. Then compare with your first answer.
A model answer
“The image is cut into 16 by 16 pixel patches, 256 of them for a 256 px image. Each patch's 768 pixel values are mapped to a 384-number token. Position information is added and a class token and four storage tokens join the sequence. Stacked attention layers let every token blend information from all the others. At the end the patch tokens are reshaped into a 16 by 16 grid of 384-number vectors, which is an explicit, lookup-read representation. A 512 px input gives 32 by 32 tokens at sixteen times the attention cost.”
Something unclear, or want to rehearse how you would say this to a reviewer? Ask your teacher in the chat.