Part II · The teacher signal · Chapter 6

How a vision transformer sees an image

About 18 minutes. Part I built a representation that can carry features. Part II explains where the features come from. This chapter covers the network that produces them.

6.1 Why the teacher matters

Everything your Gaussians store is copied from a teacher: a frozen network that reads the image and writes a 384-number vector for every patch. The error you report is the distance between your field and the teacher's field. If you cannot say what the teacher's field is, you cannot say what your number measures.

By the end you can describe, in a few sentences, how a vision transformer turns a 256 px image into a 16×16 grid of 384-number vectors, why a 512 px image gives 32×32, and why that grid is an explicit representation in Chapter 1's sense.

First, write down what you think happens between “image goes in” and “feature map comes out”. A guess is fine.

6.2 Cut the image into patches

A vision transformer, or ViT, does not look at pixels one at a time. It cuts the image into small squares and treats each square as one item, the way a language model treats a word. The original paper made this the title: an image is worth 16×16 words[Dosovitskiy et al., 2020]. It showed that a pure Transformer applied to sequences of image patches does well on image classification.

Patch. A square block of pixels, here 16 by 16. Patch size p = 16. An image of side S pixels gives G = S / p patches along each side and G² patches in all.
Input side SPatches per side GPatches G²
256 px16256
512 px321024

Your teacher is the DINOv3 ViT-S with patch size 16. Doubling the side of the image multiplies the number of patches by four. Look at what that means for the grid:

256 px: 16×16 = 256 tokens 512 px: 32×32 = 1024 tokens

Both panels show the same picture. Each small square is one token. On the right the squares are half as wide on the picture, because the network still reads 16 px squares but the image has been enlarged to twice the side. Chapter 8 returns to what the left grid hides.

6.3 From a square of pixels to a token

A 16×16 patch of colour pixels holds 16 × 16 × 3 = 768 numbers. A learned linear map turns those 768 numbers into one vector of the network's working width. For the ViT-S that width is 384, the same D = 384 you met in Chapter 1.

Token. One vector of D numbers standing for one item in the sequence. After the first step, a token for a patch is just a 384-number vector; the network no longer sees a square.

In this project frozen_dino_runtime_grid.py shows the first step plainly: a pack kernel rearranges the image into G² rows of 768 normalised pixel values, one row per patch, ready for that linear map. The pixel values are first shifted and scaled by the standard ImageNet colour statistics.

Nothing so far says where a patch was. A bag of 256 vectors has no order. The next step fixes that.

6.4 Position and the class token

Attention (next step) treats its input as an unordered set. To let the network know that one patch sits above another, the model injects position information. Classic ViTs add a learned vector for each grid position to the token. Your teacher uses a related scheme called rotary position embedding: position is mixed into the attention calculation itself. The code comment says the coordinates are normalised per axis as (i + 0.5) / G × 2 − 1, which is the same [−1, 1] frame your Gaussians use.

Rotary embeddings work for any grid size, because position is computed from coordinates rather than looked up in a fixed table. That is why one set of weights can read both a 256 px and a 512 px image. A learned table has one entry per grid cell and would need resizing.

Two kinds of extra tokens travel with the patch tokens:

  • A class token (CLS). One extra vector that belongs to no patch. Through attention it gathers information from the whole image and is read out at the end as a summary of the picture. Your cache stores it as the global feature.
  • A few storage tokens, four in this teacher. Chapter 7 explains why they exist.

The teacher's token sequence for a 256 px image is therefore 1 + 4 + 256 = 261 tokens, and for a 512 px image 1 + 4 + 1024 = 1029. The code comment in frozen_dino_runtime_grid.py gives the 261.

6.5 Attention in one paragraph

A transformer is a stack of identical layers. Inside each layer every token produces three vectors of its own: a query (what it is looking for), a key (what it offers) and a value (what it would hand over). A token compares its query with every token's key by dot product, turns the scores into weights that sum to one with a softmax, and takes the weighted average of the values. The result is added back to the token. Many such averages run in parallel (called heads), and a small feed-forward network follows. After many layers each patch token has mixed in information from every other patch, so the vector for a patch describes the patch in the context of the whole image.

Attention. A learned, content-dependent weighted average: each token replaces itself by a blend of all tokens' values, with blend weights that come from query-key similarity.

Two consequences matter for your work.

  • Cost. Every token compares with every other, so the number of pairs is the token count squared. At 256 tokens that is 65,536 pairs; at 1024 tokens it is 1,048,576, sixteen times as many. Four times the tokens costs sixteen times the attention pairs.
  • Features are contextual. A patch of sky and a patch of sky next to a bird do not get identical vectors. The teacher field is not a local filter of the image.

6.6 The output is a grid of vectors

After the last layer the code does a final normalisation and then extracts the patch tokens and the CLS token. Discard the storage tokens, reshape the G² patch tokens back to G × G, and you have the teacher field.

Feature map (teacher field). A G × G grid with a 384-number vector in each cell. In Chapter 1's language it is an explicit representation: values at grid points, read by lookup. It is a signal f : Ω → ℝ³⁸⁴ sampled on a coarse grid.

Chapter 4 fitted Gaussian features to exactly this object: the target matrix T has one row per grid cell. Two small details in this project's pipeline:

  • The image is resized square with bicubic interpolation, with no crop, so the whole picture is always in view (cache_frozen_patches_fine_20261113.py).
  • Patch tokens are stored as 16-bit numbers in the cache, and the models normalise each token to unit length themselves. The cache does not.

The grid is a result, not a choice you made. The teacher decides the sampling pattern: one vector per 16×16 pixels. To get a finer grid you must change the input, not the Gaussians. That is the subject of Chapter 8.

6.7 Check yourself

1. A 512 px image goes through a patch-size-16 ViT. How many patch tokens result?

Side 512 / 16 = 32 patches per side, so 32 × 32 = 1024. Doubling the side quadruples the count.

2. Why does the network need position information at all?

Attention is a weighted average over a set. Without a position signal, shuffling the patches would give the same result for each token's content.

3. Why is the 16×16 feature map an explicit representation?

Explicit means stored values read by lookup. Who computed them is irrelevant, and a grid is defined only at its grid points.

4. Why is a patch token more than a description of its own square?

After many attention layers each token carries context from the whole image. The cache does not normalise, and attention is global rather than local.

5. Going from 256 px to 512 px multiplies the attention pair count by what?

Every token compares with every token, so pairs scale with tokens squared: 4² = 16.

6.8 Close the loop

Without scrolling up, explain how a 256 px image becomes the teacher field your Gaussians fit. Then compare with your first answer.

A model answer

“The image is cut into 16 by 16 pixel patches, 256 of them for a 256 px image. Each patch's 768 pixel values are mapped to a 384-number token. Position information is added and a class token and four storage tokens join the sequence. Stacked attention layers let every token blend information from all the others. At the end the patch tokens are reshaped into a 16 by 16 grid of 384-number vectors, which is an explicit, lookup-read representation. A 512 px input gives 32 by 32 tokens at sixteen times the attention cost.”

Read next Dosovitskiy et al., “An Image is Worth 16x16 Words” (2020), the method section on how an image becomes a sequence of patch embeddings. The paper is short and is the one the whole field cites for this step.

Something unclear, or want to rehearse how you would say this to a reviewer? Ask your teacher in the chat.

One-page summary of this chapter ·