A short textbook

Semantic Gaussian Fields

The ideas behind representing an image's meaning with a few Gaussians, built up one chapter at a time. Each chapter assumes only the chapters before it.

The map

Where this work sits. Circles: representing a signal (Chapters 1 to 5, 15), foundation-model features (Chapters 6 to 8, 17), tokens and grouping (Chapter 16). Overlaps: feature fields (17), Gaussian tokens (16), slots over DINO (10, 16). Boxes: clustering (9, 10), least squares (4), learned optimisation (11, 12), budgets (13), training tools (18).

Venn diagram of the three fields this work sits between

Part I · Representing a signal

  1. Signals and how to store them
  2. The 2D Gaussian
  3. Many Gaussians: summing, compositing and sharing
  4. Giving each Gaussian a feature: least squares
  5. Putting it together: is this a neural representation?

Part II · The teacher signal

  1. How a vision transformer sees an image
  2. Self-supervised features: from DINO to DINOv3
  3. Resolution: why a 16×16 grid is coarse

Part III · Placing the Gaussians

  1. Clustering: k-means and EM
  2. Learning to cluster
  3. Optimisation inside the network
  4. The amortisation gap

Part IV · How many Gaussians?

  1. Budgets and description length
  2. Reading the negative results

Part V · The landscape

  1. Gaussian splatting in 3D and 2D
  2. Tokenization beyond patches
  3. Feature fields

Part VI · Doing it properly

  1. Optimisers and kernels
  2. Evaluation and rigour
  3. Positioning the work

Appendix

Reference

To read offline: open this page once, then use the browser menu, Add to Home screen (Android: Install app).