Part III · Placing the Gaussians · Chapter 12

The amortisation gap

About 20 minutes. Tap Continue to move on; nothing later depends on anything you have not yet seen.

12.1 How far from the best possible?

Chapter 1 called your placement amortised: a network trained once outputs the parameters for a new image in one pass, instead of an optimisation per image. Every amortised method invites the same question from a reviewer: how much worse is your one pass than optimising each image properly? The answer has a name, a literature and a correct way to measure it. This chapter gives all three, then turns your Part III results into one ladder you can present.

By the end you can define the amortisation gap and its partner, the approximation gap; place your model in Amos's vocabulary (semi-amortised, objective-based); read your ladder on both targets; and state “how close to the optimum” in a way that survives scrutiny.

Before reading on: the roadmap says the trained model is “15% above the per-image optimum” on the 32×32 target. What would you want to know before believing that number?

12.2 Two gaps

The term comes from variational autoencoders, where an encoder network predicts, for each input, the parameters of an approximate distribution. Cremer, Li and Duvenaud split the shortfall of that prediction into two parts[Cremer et al., 2018]: an approximation gap, because the chosen family of distributions cannot match the truth even at its best member; and an amortisation gap, “the difference caused by amortizing the variational parameters over the entire training set, instead of optimizing for each training example individually”. They found the shortfall was usually due to the encoder, not the family.

Translate to your setting, with error in place of likelihood:

Approximation gap. The dense error of the best placement of K Gaussians for an image (with least-squares features): what the family cannot represent at all.
Amortisation gap. The error of the network's placement minus that best error: what the one-pass prediction gives away.

The two respond to different levers. More Gaussians shrink the approximation gap (at 16×16, 256 Gaussians fit the 256 tokens exactly, error 6×10−12; Chapter 13 covers the count). A better predictor shrinks only the amortisation gap. Parts I and II fixed the family; Part III has been about the second gap all along.

12.3 Amos's vocabulary

Amos's tutorial treats any “learned model that predicts the solution of an optimisation problem” as amortised optimisation, and sorts methods by two choices[Amos, 2022]:

  • Model: fully or semi-amortised. A fully-amortised model is a black box from input to solution, with no access to the objective. A semi-amortised model “accesses the objective”, typically by taking optimisation steps on it inside the model.
  • Loss: regression- or objective-based. Regression-based learning fits the model's output to precomputed ground-truth solutions. Objective-based learning trains the model on the objective's value at its output, with no ground-truth solutions.

Your ladder walks across the first choice:

ModelAccesses the objective?Amos's class
Gridless64 (Chapter 10's loop, gentle knobs)NoFully amortised
Clustering BOnly through residual weightsFully amortised, in effect
Refinement JYes: 16 gradient steps on itSemi-amortised

All are trained objective-based: on the dense error, never regressed onto per-image optimal layouts. Regression onto detached per-image refinements was on the roadmap (phase 2b) as an option and was not built.

Two close precedents for “network start plus optimisation steps”: semi-amortised VAEs, where an inference network initialises the variational parameters, stochastic variational inference refines them, and the refinement is differentiated so the whole system trains end to end[Kim et al., 2018]; and iterative amortised inference, where a learned model refines the estimate repeatedly[Marino et al., 2018]. Your difference: the steps are on a reconstruction error with a closed-form inner solve, over a set of geometric primitives, and the step direction is detached.

12.4 The ladder on the 16×16 target

64 Gaussians, least-squares features, one seed; trained rungs on 800 holdout images.

RungDense MSE× optimumGap closed
One shot (least-squares Gridless64)0.0001042.240%
Unrolled clustering (B)0.00008041.7341%
+ 16 refinement steps (J)0.00005871.2679%
+ 32 refinement steps (J run on)0.00005311.1488%
Per-image optimum (100 images)0.00004651100%

“Gap closed” is the share of the distance from the one-shot rung to the optimum that each rung covers. The optimum here is 600 Adam steps from a grid start on the first 100 images, so the last two columns mix cohorts.

A useful way to read J: per-image Adam from a grid start (same 100 images) needs about 100 steps to reach 0.0000576. J reaches 0.0000587 within a single forward pass that contains 16 steps. That comparison is across cohorts and starts, so use it as an illustration of what amortisation buys, not as a measured speed-up.

12.5 What “the optimum” really is

The objective is not convex. The per-image “optimum” is the best result an optimiser found, and it depends on where it started and how long it ran:

Per-image optimisation (16×16, 64 Gaussians)Dense MSE
600 Adam steps from a wide 8×8 grid0.0000465
200 Adam steps from per-image k-means0.0000483
600 Adam steps from the Gridless64 layout0.0000558

Quick experiments, plain PyTorch, first 100 holdout images; s4 and s5. Step sizes were set in advance (s4) or tuned on the same images (s5).

Three consequences for how you talk about it:

  1. The true best placement is at least as good as 0.0000465, so the measured amortisation gap is a lower bound on the true one. Say “the best per-image result we found”.
  2. Starts matter a great deal (0.0000465 against 0.0000558 for the same 600 steps). That is the same lesson as k-means in Chapter 9, and it is why a learned start is worth having.
  3. Always name the reference: optimiser, steps, start, image count.

12.6 The ladder on the 32×32 target, and the “15%”

On the input512 target (Chapter 8), model J at 64 Gaussians: 0.0001526 before refinement, 0.0001137 after 16 steps, 0.0001062 after 32 (800 holdout images, one seed). The reference here is s7: per-image k-means followed by 16 or by 200 Adam steps, on the first 100 holdout images, with the clustering width tuned on those images.

On the same first 100 imagesDense MSE× 200-step result
Per-image k-means, no steps0.0001491.51
Per-image k-means + 16 steps0.0001171.18
J, 16 steps0.0001121.13
J, 32 steps0.0001041.05
Per-image k-means + 200 steps0.00009881

Now the roadmap's “15% above the per-image optimum”: it is J's 800-image value (0.0001137) divided by the 100-image reference (0.0000988). On matched images the figure is 13% at 16 steps and 5% at 32 steps. The difference is small here, but the habit matters: a ratio of two numbers from different image sets can move by more than the effect you claim. Note also that J at 16 steps is slightly better than per-image k-means followed by the same 16 steps (ratio 0.96), and that refinement had not levelled off at 32 steps.

The gap grows with the number of Gaussians (32% above at 256 with 32 steps). Chapter 13 covers that.

12.7 How to present it

A sentence that survives a hostile reader:

“On the 32×32 DINOv3 target with 64 Gaussians, our feed-forward predictor ends 13% above the best per-image result we found (200 Adam steps from k-means) after 16 internal refinement steps, and 5% above after 32, measured on the same 100 holdout images; one seed, with the k-means width tuned on those images.”

The checklist behind it:

  • Same images, same metric for every ratio.
  • Name the reference (optimiser, start, steps) and call it the best found, not the optimum.
  • Give the curve, not one point: error against refinement steps, beside the per-image curve, as one figure (roadmap P5). It shows the gap closing as steps grow.
  • Give both readings: “× optimum” and “share of the gap closed from the one-shot predictor”; they answer different questions.
  • State the caveats: one seed in development, 100 versus 800 images, settings tuned on the evaluation images, holdout reused; the sealed test set is still to come.
  • Position it: no paper found studies the amortisation gap for reconstruction with a variable number of primitives (RESOURCES.md, gaps). Measured carefully, this analysis is a contribution in its own right.

12.8 Check yourself

Answer from memory; each question gives its reason once you choose.

1. What is the amortisation gap in this project?

The first is the approximation gap, the third a generalisation gap. The amortisation gap is what one-pass prediction gives away against per-image optimisation in the same family.

2. Why is the “per-image optimum” only an estimate?

600 steps give 0.0000465 from a grid and 0.0000558 from the Gridless64 layout. The ridge solve is unique and the holdout is fixed.

3. In Amos's terms, model J is which?

J takes gradient steps on the objective inside the model (semi-amortised) and is trained on the dense error, never on precomputed optimal layouts (objective-based).

4. What is wrong with “15% above the optimum” as first stated?

0.0001137 is on 800 images and 0.0000988 on the first 100. On matched images it is 13% at 16 steps and 5% at 32.

5. “Per-image optimisation is better, so why amortise?” Best answer?

J closes about 79% (16 steps) to 88% (32 steps) of the one-shot gap on 16×16 in a single pass, and starts matter for per-image optimisation too. Per-image runs also use least squares, and they remain lower.

12.9 Close the loop

Without scrolling up: define the two gaps, place your model in Amos's vocabulary, and give your headline gap number with its caveats. Then compare with your first answer.

A model answer

“The approximation gap is what the best placement of K Gaussians still misses; more Gaussians shrink it. The amortisation gap is our network's error minus the best per-image error; a better predictor shrinks it. Our model is semi-amortised and objective-based: a learned clustering start, then 16 Adam steps on the true error inside the network, trained on the dense error. On 16×16 the ladder goes 0.000104, 0.0000804, 0.0000587, 0.0000531 against a best-found 0.0000465 on 100 images. On 32×32, on the same 100 images, we end 13% above a 200-step per-image result after 16 steps and 5% after 32. One seed, settings tuned on those images, and the ‘optimum’ is the best we found, so the true gap is at least this large.”

Read next Amos, “Tutorial on amortized optimization” (2022), chapter 2 (sections 2.1 and 2.2). The fully/semi-amortised and regression/objective-based distinctions this chapter used, with examples from other fields.

Something unclear, or want to rehearse how you would say this to a reviewer? Ask your teacher in the chat.

One-page summary of this chapter ·