Summary · Chapter 6

How a vision transformer sees an image

Pipeline: cut the image into 16×16 px patches; map each patch's 768 pixel values to a 384-number token; add position information; add a class token and four storage tokens; stack attention layers; drop the extras and reshape the patch tokens into a G × G grid, G = S / 16.

Input side SGridPatch tokensAll tokens
256 px16×16256261
512 px32×3210241029

Attention: each token replaces itself by a weighted average of all tokens' values, with weights from query-key similarity (softmax of dot products). Cost grows with tokens squared: 4× the tokens, 16× the pairs.

Output: a grid of 384-number vectors, an explicit representation (values at grid points, read by lookup). Position comes from rotary embeddings in this teacher, so any multiple of 16 works as input size.

Defence answer: “The teacher field is one 384-number vector per 16×16 pixel patch, computed with global context; the grid is the teacher's sampling, not mine.”

Source: Dosovitskiy et al., An Image is Worth 16x16 Words (2020). Code: frozen_dino_runtime_grid.py.