How a vision transformer sees an image
Pipeline: cut the image into 16×16 px patches; map each patch's 768 pixel values to a 384-number token; add position information; add a class token and four storage tokens; stack attention layers; drop the extras and reshape the patch tokens into a G × G grid, G = S / 16.
| Input side S | Grid | Patch tokens | All tokens |
|---|---|---|---|
| 256 px | 16×16 | 256 | 261 |
| 512 px | 32×32 | 1024 | 1029 |
Attention: each token replaces itself by a weighted average of all tokens' values, with weights from query-key similarity (softmax of dot products). Cost grows with tokens squared: 4× the tokens, 16× the pairs.
Output: a grid of 384-number vectors, an explicit representation (values at grid points, read by lookup). Position comes from rotary embeddings in this teacher, so any multiple of 16 works as input size.
Defence answer: “The teacher field is one 384-number vector per 16×16 pixel patch, computed with global context; the grid is the teacher's sampling, not mine.”
Source: Dosovitskiy et al., An Image is Worth 16x16 Words (2020). Code: frozen_dino_runtime_grid.py.