16.1 Why this chapter matters
Your Gaussians are a way of cutting an image into a small number of pieces. Computer vision has several other ways of doing that, and each has its own literature: merging tokens, choosing patch sizes, learning tokenisers, and grouping features into slots. A reviewer who works in any of those fields will ask what yours does that theirs does not.
Before reading: in a ViT (Chapter 6) the number of tokens is fixed by the patch size. Name one reason you might want fewer tokens than the patch grid gives.
16.2 Fewer tokens for an existing ViT
Attention cost in a ViT grows with the number of tokens (Chapter 6). The simplest way to save compute is to keep the trained network and shrink the token set as it runs.
- Token merging (ToMe) repeatedly merges the most similar tokens inside the network, with no retraining. Its abstract reports roughly doubled throughput for ViT-L at 512 px and ViT-H at 518 px with a 0.2 to 0.3% accuracy drop[Bolya et al., 2022].
- DynamicViT learns to prune unimportant tokens[Rao et al., 2021].
- A-ViT lets each token halt when the network has seen enough of it, the adaptive-computation idea of Chapter 13 applied per token[Yin et al., 2022].
Shared with your project: the aim of a small, adjustable count, and the evaluation question “how good is the result at equal count?”. Different: these methods keep the original patch tokens (or averages of them) and have no geometry, no continuous read-out and no feature solve. A merged token has no location at which you can query the field.
Token merging down to your Gaussian count is still an open baseline in the roadmap (section 1: baselines at equal token count are unticked). Until it is run, you cannot claim to beat merging, only to be a different kind of object.
16.3 Content-aware patches and superpixels
Another route changes the cut itself. Instead of a uniform grid, choose pieces by what the image contains.
- APT uses larger patches in flat regions and smaller ones in detailed regions, chosen by an entropy measure. Its abstract reports 40% higher throughput for ViT-L and 50% for ViT-H while maintaining performance, and up to 30% faster training and inference on dense tasks[Choudhury et al., 2025]. The thresholds are set by hand (audit).
- SPiT tokenises with superpixels, regions of similar pixels, and uses position embeddings that do not depend on shape. Its abstract claims better attribution faithfulness and pixel-level granularity on zero-shot dense prediction, without numbers[Aasan et al., 2024].
- MSViT learns a gate that picks the token scale for each region[Havtorn et al., 2023]. FlexiViT trains one ViT that works at many patch sizes[Beyer et al., 2023].
Shared: content-adaptive cuts, and, for FlexiViT, one model that serves many budgets (your any-count result). Different: the pieces are still rectangles or superpixels, and each piece becomes one input token. Yours are overlapping soft regions with a shape (a covariance), and they are not an input to a classifier: they are a compact representation of a teacher's field. The matched-count comparison against APT is also still missing and is a roadmap item.
16.4 Learned tokenisers with a variable length
In image generation, a tokeniser compresses an image into a short sequence that a generator then models. Several recent ones give a variable number of tokens.
- FlexTok maps an image to an ordered 1D sequence of anywhere from 1 to 256 tokens. The order is imposed by nested dropout, which randomly truncates the sequence during training so that early tokens must carry the most. Its abstract reports generation FID below 2 for 8 to 128 tokens[Bachmann et al., 2025].
- ElasticTok varies the count with random masking[Yan et al., 2024]; ALIT allocates tokens by image complexity through recurrence[Duggal et al., 2024]; TiTok pushes to 32 tokens[Yu et al., 2024].
You met the idea in Chapter 13 as Matryoshka and nested representations. Shared: one model, many budgets, and a trade-off curve. Different: these tokens are abstract codes with no spatial meaning, and they are trained to reconstruct pixels for a generator; yours are explicit 2D primitives that you can draw, move and query at a location, and their target is a recognition-grade feature field. FlexTok's order comes from training with truncation; your count flexibility comes from training one model at 64 and using a nearest-query rule at test time (Figure 4 of the results, one seed).
16.5 Gaussian tokens, and the closest paper
Three papers use 2D Gaussians as tokens, and they are the ones a reviewer will bring up.
GViT
GViT is a supervised classifier whose input tokens are 2D Gaussians. A ViT-S-sized encoder predicts each Gaussian's position, scale, orientation, colour and opacity in one forward pass. A differentiable renderer reconstructs the image from them, and the same parameters feed a ViT backbone for classification, so the Gaussians must serve both jobs[Hernandez et al., 2025]. On ImageNet-1K, GViT-B reaches 76.9% top-1 against 79.8% for ViT-B/16 trained with the same recipe (version 2 abstract). I found no code and no venue listed.
Shared with yours: 2D Gaussians as the unit; feed-forward placement by a network; a renderer. Different:
- GViT's target is RGB plus a class label; yours is a frozen DINOv3 feature field.
- GViT's Gaussians come with predicted appearance; yours get their features by a closed-form ridge solve given geometry.
- GViT uses a fixed count (audit); yours is one model for 16 to 256.
- Placement in GViT is a single encoder pass; yours is eight unrolled clustering iterations and sixteen unrolled Adam steps on the true error.
How to read the 2.9-point gap: under an equal recipe, patch tokens still beat Gaussian tokens on classification. It says nothing about your dense error, and your linear-probe numbers (top-1 0.889 for model J on the 512 px target, against 0.921 for the teacher's class token, 800 images, one seed) use a different protocol and cannot be put beside it.
GPSToken
GPSToken splits the image into texture-homogeneous regions by an entropy-driven algorithm, encodes each region as a 2D Gaussian plus a texture feature, refines with a transformer, and renders to feature maps through a differentiable splatter. Its abstract reports 128 tokens, reconstruction FID 0.65 and generation FID 1.50[Zhang et al., 2025]. Shared: Gaussians that carry feature vectors and render to feature maps. Different: hand-coded entropy splitting versus learned clustering (and your split arms lost to clustering, Chapter 14); a generation decoder versus a frozen teacher; no least-squares feature solve.
GaussianToken and GaussianVision
GaussianToken is a Gaussian-splat image tokeniser (details from the audit)[Dong et al., 2025]. GaussianVision feeds 2DGS-compressed images to a CLIP-style model, reporting 3 to 23.5 times input compression[Omri et al., 2025]. In both, the Gaussians describe RGB.
16.6 Slots over frozen DINO features
Object-centric learning asks a network to split a scene into a few slots, each one standing for an object or region. Slot attention does it with iterative competitive attention: slots repeatedly claim input features, and the claims are normalised across slots, so each input is shared among them[Locatello et al., 2020]. If that sounds like your soft clustering of Chapter 10, it is.
DINOSAUR trains slot attention to reconstruct frozen DINO features rather than pixels. Its abstract's claim is that reconstructing self-supervised features is a sufficient signal for object-centric structure, and that it is the first unsupervised object-centric model to scale to real-world data, COCO and PASCAL VOC[Seitzer et al., 2023].
This is the closest recipe to yours: a frozen self-supervised teacher, a small set of competing explainers, reconstruction of the teacher's features. Different: slots have no geometry and no renderer; the aim is objects, while your Gaussians tile the image at a fixed budget regardless of objects; and your features come from a closed-form solve, not a decoder. A reviewer who says “this is DINOSAUR with Gaussians” is half right, and the right answer is to agree on the recipe and then name the three differences.
16.7 Shared and different: the comparison
| Work | Pieces | Count | Target | Placement |
|---|---|---|---|---|
| ToMe | Merged patch tokens | Reduced in the network | Existing ViT task | Similarity matching |
| APT | Square patches, mixed sizes | Fewer by content | Classification, dense tasks | Hand-set entropy thresholds (audit) |
| SPiT | Superpixels | Content-driven | Classification, dense | Superpixel algorithm |
| FlexTok | Ordered 1D codes | 1 to 256, one model | Pixel reconstruction, generation | Learned, no geometry |
| GViT | 2D Gaussians | Fixed (audit) | RGB plus class label | Encoder pass |
| GPSToken | Gaussian plus texture | Variable, 128 reported | Reconstruction, generation | Entropy split, then transformer |
| DINOSAUR | Slots, no geometry | Fixed K | Frozen DINO features | Slot attention |
| This project | 2D Gaussians, soft regions | 16 to 256, one model | Frozen DINOv3 feature field | Unrolled clustering and Adam |
The defensible novelty sentence from this chapter alone: the combination of a frozen-teacher feature field, closed-form features given geometry, softmax-shared 2D Gaussians and unrolled amortised placement for any count. No single paper has all four; closest-work.md records that this is a statement about the search, not a proof. Chapter 20 turns it into the final claim.
Equal-count baselines (pooling, per-image k-means, token merging) are still to run. Until then, “better than merging” is not a claim you can make.
16.8 Check yourself
Answer from memory; each question gives its reason once you choose.
1. What is the fairest reading of GViT’s 76.9% against 79.8%?
2. How does ToMe differ from your Gaussians at the root?
3. Why is DINOSAUR the closest recipe to yours?
4. What does FlexTok’s nested dropout do?
5. A reviewer says your model beats token merging. What is your honest status?
16.9 Close the loop
Without scrolling up, place your model among the five families and say, for GViT, one thing shared and three things different.
A model answer
“Token reduction (ToMe, pruning) shrinks the set inside a ViT. Content-aware patches and superpixels change the cut. Learned tokenisers (FlexTok) give ordered codes of variable length. Gaussian tokens (GViT, GPSToken) use 2D Gaussians for RGB tasks. Slots (DINOSAUR) share my recipe of a frozen teacher and soft competition, without geometry. Mine combines a feature-field target, closed-form features, softmax sharing and amortised placement for any count.”
Something unclear, or want to rehearse how you would say this to a reviewer? Ask your teacher in the chat.