2.1 Why one Gaussian deserves a chapter
Chapter 1 ended with a family of representations that store a set of simple shapes. In your work that shape is the 2D Gaussian: a soft, elliptical blob that sits somewhere in the image. Everything else in the model is built from it. If you can say exactly what one Gaussian is, which numbers describe it and why they are stored the way they are, most of the hard questions about the model become questions about how many blobs there are and where they go.
First, from memory: what numbers would you need to describe a tilted ellipse on a page? Write your guess.
2.2 One dimension first: a bump
Start on a line. A Gaussian is a bump with a centre μ (mu) and a width σ (sigma). Its shape is
g(x) = exp(−(x − μ)² / (2σ²))
At the centre the bump has height 1. Move away and it falls quickly, and how quickly depends only on the distance measured in units of σ:
One width away the value is 0.61, two widths away 0.14, three widths away 0.01. So a Gaussian is local: beyond about three widths it has effectively vanished. But it never reaches exactly zero, which matters in Chapter 3.
The density is the version used when Gaussians compete with each other, so keep that last sentence in mind.
2.3 Two dimensions: distance becomes an ellipse
In the image plane a location is a pair x = (x₁, x₂) and the centre is a pair μ = (μ₁, μ₂). The simplest 2D Gaussian has its own width along each axis, s₁ and s₂. Distance in width units becomes
ρ² = ((x₁ − μ₁)/s₁)² + ((x₂ − μ₂)/s₂)²
and the bump is exp(−ρ²/2), exactly as in 1D; ρ (rho) is the distance measured in widths. All points with the same ρ have the same value, and those points form an ellipse with half-axes ρ·s₁ and ρ·s₂. So a 2D Gaussian looks like nested ellipses that fade outwards.
That ellipse is aligned with the axes. To tilt it we need a description that handles any orientation, and that is what a covariance matrix is.
For the axis-aligned case, Σ is diagonal with entries s₁² and s₂², and the formula gives back the ρ² above. A tilted Gaussian is the same thing turned through an angle θ: Σ = R diag(s₁², s₂²) Rᵀ, where R is the rotation by θ. Five numbers then describe the whole shape and position: two for the centre, two axis lengths and an angle.
2.4 Reading a covariance: a worked example
Take
Σ = 0.050.030.030.05
The diagonal entries are the variances along x₁ and x₂. The off-diagonal 0.03 is positive, so the two coordinates move together: the ellipse leans along the diagonal. To find its axes exactly, look for directions that Σ only stretches.
Check by multiplying: Σ(1, 1) = (0.08, 0.08), so (1, 1) is an eigenvector with λ = 0.08. And Σ(1, −1) = (0.02, −0.02), so (1, −1) has λ = 0.02. The long axis runs along the diagonal with length √0.08 ≈ 0.283; the short axis is at right angles with length √0.02 ≈ 0.141.
Now see what this does to distance. Take two points both 0.2 from the centre:
- Along the long axis: ρ² = 0.04/0.08 = 0.5, value exp(−0.25) = 0.78.
- Along x₁: ρ² = 1.25 (worked out in the next step), value exp(−0.625) = 0.54.
Same distance on the page, different value. The covariance is a ruler that is longer in some directions than others.
2.5 Not every matrix is a covariance; the Cholesky factor fixes that
Suppose a network outputs the three entries of Σ directly and produces 0.05, 0.06, 0.05. Its eigenvalues are 0.11 and −0.01. A negative eigenvalue means a negative squared length: along that direction ρ² is negative, so exp(−ρ²/2) grows without limit. And det Σ is negative, so √det Σ does not exist. It is not a Gaussian at all.
Checking and repairing outputs during training is awkward. The clean fix is to store something from which Σ is built and which cannot go wrong.
Two facts make this the right storage. Any a > 0, d > 0 and any b give a valid covariance (its determinant is (ad)² > 0). And every valid covariance has exactly one such factor. So nothing is lost and nothing can break.
For the example: a = √0.05 = 0.2236, b = 0.03/a = 0.1342, d = √(0.05 − b²) = 0.1789. Try the same on the bad matrix: d² = 0.05 − 0.072 < 0, so the factorisation itself refuses it.
The factor also makes the density cheap. For an offset (δ₁, δ₂) = x − μ, set u = δ₁/a and v = (δ₂ − b u)/d. Then ρ² = u² + v² and √det Σ = a·d; no matrix inverse is needed. For the point 0.2 along x₁: u = 0.894, v = −0.671, ρ² = 1.25, as promised.
Things to try: set the two axes equal and turn the angle (nothing changes, and b = 0); make axis 2 tiny (the peak density shoots up); turn the angle to 0 and 90 degrees (the off-diagonal entry vanishes). However you move the sliders, a and d stay positive.
2.6 Density value and mass
Put the pieces together. For one Gaussian the logarithm of its density at x is
log φ(x) = −log 2π − log a − log d − ½(u² + v²)
The first three terms are the height of the peak; the last says how far x is from the centre in width units. Narrow Gaussians (small a, d) have a tall peak and fall off fast. In the example the peak is 1/(2π · 0.04) ≈ 3.98.
One more number completes a Gaussian.
On its own a mass only scales one bump up or down. It matters when several Gaussians share the image, which is Chapter 3.
2.7 This project's Gaussian: six numbers
Your model works in normalised image coordinates: the image is the square [−1, 1]², with x₂ pointing down so that row 0 is at the top (grid_points in geometry.py). The image is 2 units wide, so on a 32×32 grid one cell is 2/32 = 0.0625 wide.
The placement network outputs six unconstrained numbers per Gaussian. A decoder (_decode_free in free_gaussian_spatial.py) turns them into a valid Gaussian:
| Raw outputs | Becomes | Rule | Guarantee |
|---|---|---|---|
| 2 | centre μ | tanh of each | inside the image, (−1, 1)² |
| 2 | a, d | 0.008 + 0.05·softplus | positive, at least 0.008 |
| 1 | b | 0.05 × raw | any value is allowed |
| 1 | mass logit | kept as is | masses = softmax over Gaussians |
Softplus is a smooth function that is always positive (softplus(z) = log(1 + eᶻ)), and softmax turns a list of numbers into positive numbers that sum to one. So whatever the network outputs, the result is a valid Gaussian inside the image with masses that add up to one. With a raw output of zero, a = d ≈ 0.043, about two thirds of a 32×32 cell. The floor 0.008 is about 2 pixels at 512 pixels wide. (The later refinement stage, Part III, uses QDecode in free_gaussian_refine.py, with a = 0.004 + eʳ; same idea, a positive floor.)
The density itself is gaussian_log_density in geometry.py, and it is the formula of 2.6 line for line: zx = delta/a, zy = (delta − b·zx)/d, and log m − log 2π − log a − log d − ½(zx² + zy²). covariance_from_cholesky builds Σ exactly as in the definition box.
2.8 Check yourself
Answer from memory; each question gives its reason once you choose.
1. Why does the model store a Cholesky factor instead of the three entries of the covariance?
2. For Σ with diagonal 0.05, 0.05 and off-diagonal 0.03, where is the long axis?
3. A network outputs covariance entries 0.05, 0.06, 0.05 directly. What is wrong?
4. Two Gaussians share a centre; one has both axes 0.1, the other both axes 0.2. Compare their peak densities.
5. A reviewer asks what the six numbers per Gaussian are. Which answer is accurate?
2.9 Close the loop
Without scrolling up, explain how six numbers describe one Gaussian in your model, and why they are not simply a centre and the three entries of Σ. Then compare with your first answer.
A model answer
“A 2D Gaussian is a centre and a covariance: the covariance is a symmetric 2×2 matrix whose eigenvectors are the ellipse's axes and whose eigenvalues are the squared axis lengths. It must be positive definite, or the density is not defined. Instead of the covariance, the model stores its Cholesky factor, a lower-triangular L with positive diagonal, because any such L gives a valid Σ = LLᵀ and the density is cheap to evaluate from it. The six numbers are the centre (two, kept inside [−1, 1]² by tanh), the factor (three, diagonal kept above a floor by softplus) and a mass logit (one; a softmax makes the masses positive and sum to one).”
Something unclear, or want to rehearse how you would say this to a reviewer? Ask your teacher in the chat.