Part V · The landscape · Chapter 15

Gaussian splatting in 3D and 2D

About 20 minutes. Tap Continue to move on.

15.1 Why this chapter matters

Whenever you say “2D Gaussians that carry features”, someone will hear “Gaussian splatting”. They will ask whether this is 3DGS in two dimensions, or whether GaussianImage already did it. You need to know exactly what the splatting family does, which parts you borrowed, and which you replaced on purpose.

By the end you can explain 3D Gaussian splatting in a minute, say how image-plane Gaussians differ from it, and name four things your model does that none of the RGB image-Gaussian papers do.

Before reading: in one or two sentences, what do you think splatting adds on top of “a sum of Gaussians” (Chapter 3)?

15.2 3D Gaussian splatting in one pass

The task 3D Gaussian splatting (3DGS) solves is novel-view synthesis: given photographs of a scene from several camera positions, render it from a new position. The scene is stored as a cloud of 3D Gaussians[Kerbl et al., 2023]. Each one has a centre in space, a 3D covariance (an ellipsoid: Chapter 2, one dimension up), an opacity and a colour.

To draw an image from one camera, the renderer does the following:

  1. Project. Each 3D ellipsoid is flattened onto the image plane. The result is a 2D Gaussian, with the covariance of Chapter 2.
  2. Sort. Order the Gaussians by distance from the camera.
  3. Composite. At each pixel, walk through the overlapping Gaussians from near to far, letting each add its colour in proportion to its opacity and to how much light the nearer ones let through.

Training then compares the rendered image with the photograph and adjusts every Gaussian's parameters by gradient descent, starting from the sparse 3D points that camera calibration produces. The abstract names three ingredients: that initialisation, “interleaved optimization and density control”, and a fast visibility-aware renderer[Kerbl et al., 2023]. It reports real-time rendering (at least 30 frames per second) at 1080p. Everything is optimised per scene, so 3DGS is a hybrid representation in Chapter 1's sense: explicit parts, a simple read-out, parameters found by per-scene optimisation.

Splatting. Drawing a primitive by “throwing it” onto the image: compute where its footprint lands and add its contribution to the pixels it covers. The opposite approach, ray casting, asks each pixel which primitives lie along its ray.

15.3 Compositing versus normalised sharing

Chapter 3 introduced two ways to combine Gaussians. Splatting uses the first, in front-to-back form. For a pixel at location x, with Gaussians sorted near to far:

C(x) = Σi ci αi(x) Πj<i (1 − αj(x))

Here ci is the colour of Gaussian i and αi(x) is its opacity times its Gaussian value at x. The product is the transmittance: the fraction of light that survives the nearer Gaussians. A solid foreground Gaussian hides everything behind it.

Your model uses the second way. Responsibilities rk(x) are a softmax over Gaussians, they sum to one at every location, and the field is Σk rk(x) zk. There is no depth order and no occlusion. Every location is fully shared among the Gaussians that cover it.

3D splattingThis project 3D Gaussiansproject, sort by depthcomposite near to farRGB image 2D Gaussiansevaluate at sample pointssoftmax over Gaussiansfeature field

Chapter 3's key point still holds: with softmax sharing, a Gaussian owns a soft region, so it can carry one feature vector for that region. Compositing is about what is visible; sharing is about who explains a location.

15.4 Densification: where do the Gaussians go?

3DGS does not know in advance how many Gaussians a scene needs. Its answer is adaptive density control: during optimisation it periodically adds Gaussians where the image is poorly explained, and removes useless ones. The original recipe has three moves: clone a small Gaussian in an under-covered area, split a large Gaussian that smears over detail, and prune Gaussians that have become nearly transparent[Kerbl et al., 2023]. The trigger is a heuristic based on gradients, so it is a rule of thumb, not a derived optimum.

A large literature tries to improve that rule. Each entry below is described only at the level of the resource note, not from the full paper:

Now connect this to Chapter 14. Your split-growth arms, your merge-down arm and your adaptive-count arm were all attempts at densification-style control of the count. They lost to a simpler thing: clustering directly at the target count, then refining. That is a clean sentence for a reviewer who asks “why not densify?”. The honest limit is that you tested moment-preserving splits and merges on 100 to 200 images with settings tuned on those images, not the gradient-triggered rules from the 3D literature.

15.5 A name clash: 2DGS is not image-plane

“2D Gaussian splatting” means two different things.

  • 2DGS (Huang et al.) replaces 3D ellipsoids with flat 2D disks (surfels) inside a 3D scene, so that surfaces and their normals are well defined[Huang et al., 2024]. It is still multi-view 3D reconstruction.
  • Image-plane Gaussians represent a single 2D image with Gaussians that live in the image itself. This is what you do.

If a reviewer says “2DGS exists”, check which one they mean. The first is about surface geometry and has no bearing on your claim. The second is the next step.

15.6 Gaussians in the image plane

Once you drop the third dimension, the camera, the depth sort and the multi-view photographs disappear, and a Gaussian representation of one image remains. Four papers matter, all fitting RGB.

  • GaussianImage stores an image as 2D Gaussians with 8 parameters each (position, covariance, colour), and renders by accumulated summation instead of alpha blending. Its abstract reports at least 3 times less GPU memory and at least 5 times faster fitting than implicit neural representations, with rendering at 1500 to 2000 frames per second[Zhang et al., 2024]. The framing is compression.
  • Image-GS allocates Gaussians where the error is large, progressively, which gives a level-of-detail hierarchy. Its abstract reports 0.3K multiply-accumulates to decode one pixel[Zhang et al., 2025]. Per-image optimisation.
  • Instant GaussianImage trains a network that outputs a coarse Gaussian set, with a count that adapts to image complexity, then fine-tunes briefly. Its abstract reports training time reduced by up to one order of magnitude at comparable quality[Zeng et al., 2025].
  • Fast 2DGS uses one network to predict placement by complexity and another to regress attributes, followed by a short fine-tune[Wang et al., 2025]. This description comes from the literature audit and was not re-read in the full paper.

The last two are the closest relatives of your amortised placement, and they use the two-stage idea you met in Chapter 12: a network gives a good start, an optimiser closes the gap. Where your work differs is in what is optimised and what sits inside the model.

Levels of Gaussians for very large images[Zhu et al., 2025], octree anchors[Ren et al., 2024] and hierarchical merging[Kerbl et al., 2024] offer explicit level-of-detail hierarchies. Your any-count result is the single-model alternative: one network trained at 64 serves 16 to 256 Gaussians (Chapter 13).

15.7 Shared and different: the comparison

Four facts from this project separate it from all of the above:

  1. The target is a feature field. A 384-number DINOv3 vector at each of 32×32 locations, not three colour numbers per pixel.
  2. Features are solved, not learned. Given geometry, the ridge least-squares solve of Chapter 4 sets every Gaussian's vector in closed form. The RGB papers optimise colour by gradient descent alongside geometry.
  3. Normalised sharing. Softmax responsibilities, not alpha compositing or plain summation.
  4. Placement is inside the model. Eight unrolled soft-clustering iterations and sixteen unrolled Adam steps (Chapters 10 and 11), one model for 16 to 256 Gaussians.
WorkSpaceCombinePlacementCarries
3DGS3D scene, many viewsAlpha compositePer scene; clone, split, pruneRGB
2DGS (surfels)3D sceneRay-splat intersectionPer sceneRGB, geometry
GaussianImageOne imageSummationPer imageRGB
Image-GSOne imageNot stated on abstractPer image; error-guidedRGB
Instant-GIOne imageNot stated on abstractNetwork, then fine-tuneRGB
Fast 2DGSOne imageNot stated on abstractNetwork, then fine-tune (audit)RGB
This projectOne imageSoftmax sharingUnrolled clustering and Adam, any count384-d DINOv3 features

The combine rule for Image-GS, Instant-GI and Fast 2DGS is not on their abstract pages; Instant-GI builds on GaussianImage, so summation is the likely rule, but check before saying so. “Audit” marks details not re-read in the full paper.

15.8 Check yourself

Answer from memory; each question gives its reason once you choose.

1. What does alpha compositing do that your softmax sharing does not?

Compositing orders Gaussians by depth and lets nearer ones block farther ones. Softmax sharing has no depth and always divides each location fully among the Gaussians; both can give each Gaussian one vector.

2. A reviewer says “2D Gaussian splatting already exists”. What is the best first reply?

The name covers two things. Surfels inside 3D scenes (Huang et al.) are unrelated to your claim; image-plane Gaussians are the real neighbours. Clarify before you defend.

3. Why is GaussianImage a hybrid representation in Chapter 1’s sense?

Explicit Gaussians with a simple summation rule, parameters found per image by optimisation. No decoder network, so not implicit; not a lookup grid, so not purely explicit.

4. Which description fits Instant GaussianImage’s placement?

It predicts a coarse set with a network and then fine-tunes briefly. Yours differs because the optimiser is unrolled inside the model and features are solved in closed form.

5. Your split and merge arms lost to direct clustering. What is the careful claim?

The recorded result covers moment-preserving splits and merges on 100 to 200 images, with settings tuned on those images. The gradient-triggered 3D rules were not tested, so do not claim more.

15.9 Close the loop

Without scrolling up, explain how 3D splatting differs from your model in four points: what is stored, how Gaussians combine, how placement is found, what is carried.

A model answer

“3DGS stores a scene as 3D Gaussians, projects them, sorts by depth and alpha-composites, with clone, split and prune density control, all per scene. Image-plane Gaussian papers (GaussianImage, Image-GS, Instant-GI) drop the 3D and fit RGB. Mine fits a frozen feature field, solves features by least squares, shares locations by softmax and predicts placement with an unrolled model for any count.”

Read next Kerbl et al., “3D Gaussian Splatting for Real-Time Radiance Field Rendering” (2023), the method section on adaptive density control. Then the Hugging Face introduction if the paper feels dense.

Something unclear, or want to rehearse how you would say this to a reviewer? Ask your teacher in the chat.

One-page summary of this chapter ·