Part III · Placing the Gaussians · Chapter 11

Optimisation inside the network

About 25 minutes. Tap Continue to move on; nothing later depends on anything you have not yet seen.

11.1 Steps on the error itself

Chapter 10 left a gap. The trained clustering stage reaches 0.0000804; per-image optimisation of the same 64 Gaussians reaches about 0.000047. A clustering step moves toward moments, not down the error. To close the gap, the network must take steps on the error itself.

Before building anything, the roadmap ran a gate (quick check s5): would a few gradient steps be worth it? This chapter explains the idea of putting an optimiser inside a network, the evidence that justified it, the module you built, and the six training arms that decided its final form.

By the end you can explain unrolled optimisation and where it comes from (LISTA, learning to optimise), describe your 16 Adam steps and their two gradient paths, and explain why arms D and E failed, why J won, and why learned step sizes were dropped.

Before reading on: if the network runs 16 optimiser steps internally, what part of it is actually learned? Write a guess.

11.2 Unrolling an optimiser

Chapter 10 unrolled a clustering algorithm. You can unroll any iterative method, including an optimiser: each layer takes one step of gradient descent (or a variant) on a known objective, and the network's trainable parts are trained through all the steps.

Unrolled optimisation. A network whose layers are the steps of an optimisation algorithm on a known objective, run for a fixed number of steps, with some parts (the starting point, step sizes, or the operators themselves) trained end to end.

Three landmarks:

  • LISTA. ISTA is a simple iterative method for sparse coding. Gregor and LeCun truncated it to a fixed small depth and learned its matrices, giving fast approximations of the sparse code[Gregor and LeCun, 2010]. A survey of the many networks built this way, and of design choices such as sharing weights across steps, is [Monga et al., 2019].
  • The true model in every layer. Learned primal-dual reconstruction unrolls an optimiser for tomography with learned operators but keeps the true forward model inside each layer[Adler and Öktem, 2017]. Your refinement does the same: every step uses the true least-squares error.
  • Learning to optimise. Andrychowicz et al. replace the update rule itself with an LSTM that is trained to produce good steps[Andrychowicz et al., 2016]. A later primer surveys the field and its main risk: learned update rules can generalise poorly beyond what they were trained on[Chen et al., 2021].

Keep the distinction: LISTA and learned primal-dual learn operators; learning to optimise learns the update rule. As you will see, your final model learns neither. It learns the start, through a fixed optimiser.

11.3 Adam, and why it, not plain gradient descent

Adam[Kingma and Ba, 2014] keeps two running averages per parameter: of the gradient g and of its square.

m ← β1 m + (1 − β1) g,   v ← β2 v + (1 − β2) g² q ← q − s · m̂ / (√v̂ + ε)

Here q is a parameter, s the step size, m̂, v̂ the averages divided by 1 − βn at step n (to undo their start at zero), and β1 = 0.9, β2 = 0.999, ε = 10−8. Dividing by √v̂ rescales each parameter by its own gradient size: the first step moves every parameter by about s, whatever its units.

That matters here, because a Gaussian's parameters live on very different scales: a centre coordinate, a log width, a shear, a mass logit. Quick check s5 (plain PyTorch, first 100 holdout images, least squares in the loop) measured it. Starting from per-image k-means (0.0000880) with 200-step Adam reaching 0.0000483:

Method4 steps8 steps16 steps
Adam, best step size0.0000729 (38%)0.0000674 (52%)0.0000606 (69%)
Gradient descent, normalised step, best size(21%)(32%)(41%)

Percentages are the share of the gap between the k-means start and the 200-step result that is closed. Step sizes were tuned on the same images.

Two more results from s5 shaped the design. Resetting Adam's averages every 4 steps made 16 steps worse (0.0000681 against 0.0000606), so the averages are carried across all steps. And birth and death (remove the weakest Gaussian, split the one with most residual every 4 steps) also made it worse (0.0000671 against 0.0000606), so it was dropped from the fixed-budget design. The gate (close at least half the gap in 8 to 16 steps) was passed.

11.4 Your refinement module

free_gaussian_refine.py takes the clustering stage's geometry and runs K explicit Adam steps on it. Per Gaussian the optimised parameters are

q = (cx, cy, log(a − F), b, log(d − F), mass logit)

where (a, b, d) are the Cholesky entries of Chapter 2 and F = 0.004 is a width floor; the logs keep widths positive. At each step:

  1. Compute responsibilities, solve the ridge least-squares features (Chapter 4), and take the dense MSE against the teacher: the very number you report. The objective is 104 × Σimages MSE.
  2. Get its gradient g with respect to q from the existing Triton backward kernels, on a detached copy of q. The features are re-solved inside, so this is variable projection (Chapter 4): geometry outside, features eliminated.
  3. Update Adam's averages and take the step q ← q − st,group · direction, with one step size per step and group (centre, covariance, mass), starting at 0.03.

Two gradient paths. The update direction is treated as a constant during training. So the training loss reaches (a) the step sizes, through q − s · direction, and (b) the clustering stage, through the identity path q0 → q1 → … → qK: each step adds a constant, so a change in the start passes straight through to the end. No derivative of a derivative is ever taken.

Why detach? Differentiating through the direction would need second-order derivatives through the least-squares solve and the renderer: hand-written Triton backward-of-backward kernels and far more memory. The cost of detaching is that nothing can be learned about the direction, which is why Adam's decay rates stay fixed.

Before any training, plain Adam (0.03) on the trained clustering B, all 800 holdout images: 0.0000804 → 0.0000764 / 0.0000702 / 0.0000630 / 0.0000567 after 4 / 8 / 16 / 32 steps. Note the first step raises the error to 0.0000859: Adam's first step moves every parameter by about 0.03 at once, an overshoot that later steps repair.

11.5 Deep supervision, and why D and E failed

A loss only on the final output gives the early part of a deep chain a weak, indirect signal. Deep supervision adds losses at intermediate layers[Lee et al., 2014]. Here, the dense MSE after each step t = 0 … K−1 is added with weight

w · δK − t

beside the final dense term (weight 384) and smaller auxiliary terms. Since the inner loop already computed the gradient at every qt, these terms cost no extra rendering.

The first arms used w = 384 and discount δ = 0.5. That makes the weight on the start 384 × 0.58 = 1.5 with 8 steps (arm D) and 384 × 0.516 ≈ 0.006 with 16 (arm E). Both got worse during training: D from 0.0000734 at update 500 to 0.0000792 at 3,000; E from 0.0000657 to 0.0000708. Neither beat untrained refinement (0.0000702, 0.0000630). Their start degraded badly: its error before refinement was 0.000108 to 0.000118 for D and reached 0.000140 for E, against 0.0000804 for clustering B alone. Learned step sizes shrank from 0.03 to about 0.01.

Why. Along the identity path, the start's gradient is the sum of the gradients the loss sends to each later point. Near the end of refinement the dense gradient is already small, because the geometry is near a minimum. With the discount, the early terms, where the gradient is large, have almost no weight. So the start was trained mostly by the auxiliary terms and drifted.

The fix (arms G and H): uniform deep supervision, δ = 1, so every intermediate error counts equally and the start is told directly to be good.

One detail the code adds: only the final dense term of the training loss is weighted by each token's relevance to the teacher's CLS vector (semantic_importance: true). The refinement objective, the deep-supervision terms and the reported holdout metric are all unweighted.

11.6 The arms G to L

All on the 16×16 target, 64 Gaussians, 800 holdout images, one seed, 6,000 updates; deep supervision uniform. Values at 0, 16 and 32 refinement steps (a model trained with 16 steps runs on to 32 by reusing its last step sizes):

ArmTrained stepsStep sizesWeight per step01632
Plain Adam on Bnone0.03none0.00008040.00006300.0000567
G16fixed480.00009650.00006090.0000547
H16learned480.00008490.00006190.0000560
I32fixed240.0001080.00006420.0000571
J16fixed3840.00008850.00005870.0000531
K16fixed15360.00008860.00005880.0000532
L32fixed3840.00009120.00005920.0000535
Holdout dense MSE against refinement steps Four curves from 0 to 32 refinement steps on 800 holdout images. Plain Adam on clustering B starts at 8.04 and ends at 5.67; G starts at 9.65 and ends at 5.47; I starts at 10.78 and ends at 5.71; J starts at 8.85, reaches 5.87 at 16 steps and 5.31 at 32. The per-image optimum on 100 images is about 4.65. Units 10 to the minus 5. 5 6 7 8 9 10 11 0 8 16 24 32 refinement steps ×10⁻⁵ 16: trained length (G, J) per-image optimum ≈ 4.65 (100 images) J 5.31 G 5.47 plain 5.67 I 5.71
Holdout dense MSE (×10−5) after each refinement step, 800 images, one seed. K and L lie almost on J and are omitted; H is in the table. The optimum line is from 100 images and 600 steps, so it is not directly comparable.

Four readings:

  1. Uniform supervision stops the degradation, and its strength is the lever: G (48) to J (384) improves 16 steps from 0.0000609 to 0.0000587. I, with weight 24, lets its start drift to 0.000108 and does not beat untrained 32-step refinement.
  2. It saturates. K (1536) equals J, and training with 32 steps (L) is no better than J trained with 16 and run to 32.
  3. Learned step sizes were dropped. In H the centre and covariance steps shrank (the roadmap records about a third; at the end of training both were near 0.007, under a quarter of 0.03, while the mass steps grew to between 0.04 and 0.13, run summary), and its holdout error worsened after update 1,500. The roadmap records the fact; a likely reading, not tested, is that smaller steps lower the intermediate losses early (the overshoot of 11.4) at the price of later progress.
  4. The start learns to be a start. J's geometry before refinement (0.0000885) is worse than clustering B alone (0.0000804), yet after 16 steps it is better (0.0000587 against 0.0000630), and its curve has no first-step rise. The clustering stage has learned to produce a good starting point for these Adam steps, not a good answer by itself.

J is the chosen model. Three seeds: 0.0000588 at 16 steps and 0.0000532 at 32; 26.9% below clustering B and 43.7% below least-squares Gridless64 (three-seed means). Probe 0.868 to 0.876. Centre spread across images rose again, to 0.118 (run summary).

11.7 The defence: “so you learned an optimiser?”

Be exact, because the honest answer is stronger than the flattering one.

  • What is fixed: the update rule (Adam, decay rates 0.9 and 0.999) and, in J, the step size (0.03). Learned step sizes were tried (H, D, E) and dropped.
  • What is learned: the clustering stage that produces the starting geometry, trained through 16 steps of the fixed optimiser on the true error, with every intermediate error supervised.
  • What this buys: 16 untrained Adam steps on clustering B give 0.0000630; trained end to end, 0.0000587 (7% lower), and the model runs on to 32 steps without loss (0.0000531). Unlike a learned update rule, a fixed Adam rule does not break when run beyond its training horizon.
  • What it costs: each step renders the field, solves least squares and runs a backward pass, inside the forward pass. Inference needs gradients of the renderer.

So the accurate sentence is: a learned clustering initialiser, trained end to end through a fixed number of unrolled Adam steps on the true least-squares error, with uniform deep supervision. Chapter 12 gives this a name and measures how far it still is from the per-image optimum.

Go deeper: how the deep-supervision gradient is attached without a second render

In refine_geometry, each step computes g = ∂S/∂qt for the inner optimiser. The intermediate loss term MSEt has exactly that gradient, divided by 104 × batch. KnownGradient is an autograd function whose forward returns the already computed value and whose backward returns the stored g, scaled. The training graph receives the intermediate losses for the price of a multiplication.

11.8 Check yourself

Answer from memory; each question gives its reason once you choose.

1. Why is the Adam update direction detached during training?

Differentiating the direction needs derivatives of derivatives through the renderer and least squares. Step sizes and the clustering stage still get gradient, through q − s · direction and the identity path.

2. Why did arms D and E get worse during training?

With discount 0.5 the start's own error had weight 1.5 or 0.006, and the gradient arriving through the identity path from refined points is small. Their step sizes were learned, and shrank.

3. What happened to learned step sizes in arm H?

Centre and covariance steps fell to a quarter or less of 0.03 and H worsened after update 1,500; G with fixed steps was better at 16 and 32 steps.

4. J's geometry before refinement is worse than clustering B's, yet J ends better. Why?

J's update rule and step size are fixed; only the start is learned, through the steps. A good start for 16 Adam steps need not be a good answer on its own. Both use the same 800 images.

5. Which sentence describes model J accurately?

The update rule is plain Adam with fixed settings; training shapes the clustering stage that starts it, through the unrolled steps.

11.9 Close the loop

Without scrolling up: explain the refinement module, its two gradient paths, and the one change that turned failing arms into the best model. Then compare with your first answer.

A model answer

“After clustering, the network runs 16 Adam steps on the Gaussians' geometry, each on the true dense error with features re-solved by ridge least squares. The update direction is detached, so there are no second derivatives; the loss reaches the step sizes through the update and the clustering stage through the identity path from the refined geometry back to the start. With discounted deep supervision the start got almost no dense signal and drifted (D, E). Uniform supervision, weighted as strongly as the final loss, fixed it: J reaches 0.0000587 at 16 steps and 0.0000531 at 32, three seeds agree, and it saturates (K, L). Learned step sizes shrank and hurt, so the optimiser is fixed and only its starting point is learned.”

Read next Monga, Li and Eldar, “Algorithm Unrolling” (2019), sections I to III. How LISTA started the field and the design choices (shared or per-step parameters, what to learn) that your arms explored.

Something unclear, or want to rehearse how you would say this to a reviewer? Ask your teacher in the chat.

One-page summary of this chapter ·