Part V · The landscape · Chapter 16

Tokenization beyond patches

About 20 minutes. Tap Continue to move on.

16.1 Why this chapter matters

Your Gaussians are a way of cutting an image into a small number of pieces. Computer vision has several other ways of doing that, and each has its own literature: merging tokens, choosing patch sizes, learning tokenisers, and grouping features into slots. A reviewer who works in any of those fields will ask what yours does that theirs does not.

By the end you can place your model among five families of “fewer, smarter tokens”, say what each shares with yours, and give the fair reading of the one paper closest to yours.

Before reading: in a ViT (Chapter 6) the number of tokens is fixed by the patch size. Name one reason you might want fewer tokens than the patch grid gives.

16.2 Fewer tokens for an existing ViT

Attention cost in a ViT grows with the number of tokens (Chapter 6). The simplest way to save compute is to keep the trained network and shrink the token set as it runs.

  • Token merging (ToMe) repeatedly merges the most similar tokens inside the network, with no retraining. Its abstract reports roughly doubled throughput for ViT-L at 512 px and ViT-H at 518 px with a 0.2 to 0.3% accuracy drop[Bolya et al., 2022].
  • DynamicViT learns to prune unimportant tokens[Rao et al., 2021].
  • A-ViT lets each token halt when the network has seen enough of it, the adaptive-computation idea of Chapter 13 applied per token[Yin et al., 2022].
Token budget. The number of tokens a downstream model is allowed to see. Your Gaussian count is a token budget in this sense, with one difference: the tokens here are fitted to a field, not selected from the input.

Shared with your project: the aim of a small, adjustable count, and the evaluation question “how good is the result at equal count?”. Different: these methods keep the original patch tokens (or averages of them) and have no geometry, no continuous read-out and no feature solve. A merged token has no location at which you can query the field.

Token merging down to your Gaussian count is still an open baseline in the roadmap (section 1: baselines at equal token count are unticked). Until it is run, you cannot claim to beat merging, only to be a different kind of object.

16.3 Content-aware patches and superpixels

Another route changes the cut itself. Instead of a uniform grid, choose pieces by what the image contains.

  • APT uses larger patches in flat regions and smaller ones in detailed regions, chosen by an entropy measure. Its abstract reports 40% higher throughput for ViT-L and 50% for ViT-H while maintaining performance, and up to 30% faster training and inference on dense tasks[Choudhury et al., 2025]. The thresholds are set by hand (audit).
  • SPiT tokenises with superpixels, regions of similar pixels, and uses position embeddings that do not depend on shape. Its abstract claims better attribution faithfulness and pixel-level granularity on zero-shot dense prediction, without numbers[Aasan et al., 2024].
  • MSViT learns a gate that picks the token scale for each region[Havtorn et al., 2023]. FlexiViT trains one ViT that works at many patch sizes[Beyer et al., 2023].

Shared: content-adaptive cuts, and, for FlexiViT, one model that serves many budgets (your any-count result). Different: the pieces are still rectangles or superpixels, and each piece becomes one input token. Yours are overlapping soft regions with a shape (a covariance), and they are not an input to a classifier: they are a compact representation of a teacher's field. The matched-count comparison against APT is also still missing and is a roadmap item.

16.4 Learned tokenisers with a variable length

In image generation, a tokeniser compresses an image into a short sequence that a generator then models. Several recent ones give a variable number of tokens.

  • FlexTok maps an image to an ordered 1D sequence of anywhere from 1 to 256 tokens. The order is imposed by nested dropout, which randomly truncates the sequence during training so that early tokens must carry the most. Its abstract reports generation FID below 2 for 8 to 128 tokens[Bachmann et al., 2025].
  • ElasticTok varies the count with random masking[Yan et al., 2024]; ALIT allocates tokens by image complexity through recurrence[Duggal et al., 2024]; TiTok pushes to 32 tokens[Yu et al., 2024].

You met the idea in Chapter 13 as Matryoshka and nested representations. Shared: one model, many budgets, and a trade-off curve. Different: these tokens are abstract codes with no spatial meaning, and they are trained to reconstruct pixels for a generator; yours are explicit 2D primitives that you can draw, move and query at a location, and their target is a recognition-grade feature field. FlexTok's order comes from training with truncation; your count flexibility comes from training one model at 64 and using a nearest-query rule at test time (Figure 4 of the results, one seed).

16.5 Gaussian tokens, and the closest paper

Three papers use 2D Gaussians as tokens, and they are the ones a reviewer will bring up.

GViT

GViT is a supervised classifier whose input tokens are 2D Gaussians. A ViT-S-sized encoder predicts each Gaussian's position, scale, orientation, colour and opacity in one forward pass. A differentiable renderer reconstructs the image from them, and the same parameters feed a ViT backbone for classification, so the Gaussians must serve both jobs[Hernandez et al., 2025]. On ImageNet-1K, GViT-B reaches 76.9% top-1 against 79.8% for ViT-B/16 trained with the same recipe (version 2 abstract). I found no code and no venue listed.

Shared with yours: 2D Gaussians as the unit; feed-forward placement by a network; a renderer. Different:

  • GViT's target is RGB plus a class label; yours is a frozen DINOv3 feature field.
  • GViT's Gaussians come with predicted appearance; yours get their features by a closed-form ridge solve given geometry.
  • GViT uses a fixed count (audit); yours is one model for 16 to 256.
  • Placement in GViT is a single encoder pass; yours is eight unrolled clustering iterations and sixteen unrolled Adam steps on the true error.

How to read the 2.9-point gap: under an equal recipe, patch tokens still beat Gaussian tokens on classification. It says nothing about your dense error, and your linear-probe numbers (top-1 0.889 for model J on the 512 px target, against 0.921 for the teacher's class token, 800 images, one seed) use a different protocol and cannot be put beside it.

GPSToken

GPSToken splits the image into texture-homogeneous regions by an entropy-driven algorithm, encodes each region as a 2D Gaussian plus a texture feature, refines with a transformer, and renders to feature maps through a differentiable splatter. Its abstract reports 128 tokens, reconstruction FID 0.65 and generation FID 1.50[Zhang et al., 2025]. Shared: Gaussians that carry feature vectors and render to feature maps. Different: hand-coded entropy splitting versus learned clustering (and your split arms lost to clustering, Chapter 14); a generation decoder versus a frozen teacher; no least-squares feature solve.

GaussianToken and GaussianVision

GaussianToken is a Gaussian-splat image tokeniser (details from the audit)[Dong et al., 2025]. GaussianVision feeds 2DGS-compressed images to a CLIP-style model, reporting 3 to 23.5 times input compression[Omri et al., 2025]. In both, the Gaussians describe RGB.

16.6 Slots over frozen DINO features

Object-centric learning asks a network to split a scene into a few slots, each one standing for an object or region. Slot attention does it with iterative competitive attention: slots repeatedly claim input features, and the claims are normalised across slots, so each input is shared among them[Locatello et al., 2020]. If that sounds like your soft clustering of Chapter 10, it is.

DINOSAUR trains slot attention to reconstruct frozen DINO features rather than pixels. Its abstract's claim is that reconstructing self-supervised features is a sufficient signal for object-centric structure, and that it is the first unsupervised object-centric model to scale to real-world data, COCO and PASCAL VOC[Seitzer et al., 2023].

This is the closest recipe to yours: a frozen self-supervised teacher, a small set of competing explainers, reconstruction of the teacher's features. Different: slots have no geometry and no renderer; the aim is objects, while your Gaussians tile the image at a fixed budget regardless of objects; and your features come from a closed-form solve, not a decoder. A reviewer who says “this is DINOSAUR with Gaussians” is half right, and the right answer is to agree on the recipe and then name the three differences.

16.7 Shared and different: the comparison

WorkPiecesCountTargetPlacement
ToMeMerged patch tokensReduced in the networkExisting ViT taskSimilarity matching
APTSquare patches, mixed sizesFewer by contentClassification, dense tasksHand-set entropy thresholds (audit)
SPiTSuperpixelsContent-drivenClassification, denseSuperpixel algorithm
FlexTokOrdered 1D codes1 to 256, one modelPixel reconstruction, generationLearned, no geometry
GViT2D GaussiansFixed (audit)RGB plus class labelEncoder pass
GPSTokenGaussian plus textureVariable, 128 reportedReconstruction, generationEntropy split, then transformer
DINOSAURSlots, no geometryFixed KFrozen DINO featuresSlot attention
This project2D Gaussians, soft regions16 to 256, one modelFrozen DINOv3 feature fieldUnrolled clustering and Adam

The defensible novelty sentence from this chapter alone: the combination of a frozen-teacher feature field, closed-form features given geometry, softmax-shared 2D Gaussians and unrolled amortised placement for any count. No single paper has all four; closest-work.md records that this is a statement about the search, not a proof. Chapter 20 turns it into the final claim.

Equal-count baselines (pooling, per-image k-means, token merging) are still to run. Until then, “better than merging” is not a claim you can make.

16.8 Check yourself

Answer from memory; each question gives its reason once you choose.

1. What is the fairest reading of GViT’s 76.9% against 79.8%?

It is a supervised classification comparison under one recipe, ViT-B/16 ahead by 2.9 points. It does not transfer to a frozen-teacher field fit, and it says nothing about your numbers.

2. How does ToMe differ from your Gaussians at the root?

ToMe merges similar patch tokens inside a trained ViT without retraining. A merged token keeps no centre or covariance, so there is nothing to query between tokens.

3. Why is DINOSAUR the closest recipe to yours?

Frozen self-supervised features as the target, with a few competing explainers, is shared. Slots have no geometry or renderer, and its aim is objects, not a tiling.

4. What does FlexTok’s nested dropout do?

Random truncation of the sequence during training forces early tokens to carry the most information, so any prefix length gives a usable code. It is the tokeniser version of Matryoshka training.

5. A reviewer says your model beats token merging. What is your honest status?

The roadmap lists token merging at equal count as an unticked baseline. Say so, and describe what the experiment would show.

16.9 Close the loop

Without scrolling up, place your model among the five families and say, for GViT, one thing shared and three things different.

A model answer

“Token reduction (ToMe, pruning) shrinks the set inside a ViT. Content-aware patches and superpixels change the cut. Learned tokenisers (FlexTok) give ordered codes of variable length. Gaussian tokens (GViT, GPSToken) use 2D Gaussians for RGB tasks. Slots (DINOSAUR) share my recipe of a frozen teacher and soft competition, without geometry. Mine combines a feature-field target, closed-form features, softmax sharing and amortised placement for any count.”

Read next Hernandez et al., “GViT: Representing Images as Gaussians for Visual Recognition” (2025, v2), read fully. It is the paper to know best before any novelty claim.

Something unclear, or want to rehearse how you would say this to a reviewer? Ask your teacher in the chat.

One-page summary of this chapter ·