Part I · Representing a signal · Chapter 2

The 2D Gaussian

About 20 minutes, with one widget to play with. Tap Continue to move on.

2.1 Why one Gaussian deserves a chapter

Chapter 1 ended with a family of representations that store a set of simple shapes. In your work that shape is the 2D Gaussian: a soft, elliptical blob that sits somewhere in the image. Everything else in the model is built from it. If you can say exactly what one Gaussian is, which numbers describe it and why they are stored the way they are, most of the hard questions about the model become questions about how many blobs there are and where they go.

By the end you can read a 2×2 covariance matrix as an ellipse, explain why a covariance must be positive definite, and say why your model stores each Gaussian as six numbers: a centre, a Cholesky factor and a mass logit.

First, from memory: what numbers would you need to describe a tilted ellipse on a page? Write your guess.

2.2 One dimension first: a bump

Start on a line. A Gaussian is a bump with a centre μ (mu) and a width σ (sigma). Its shape is

g(x) = exp(−(x − μ)² / (2σ²))

At the centre the bump has height 1. Move away and it falls quickly, and how quickly depends only on the distance measured in units of σ:

μμ+σμ+2σμ+3σ 10.610.140.01

One width away the value is 0.61, two widths away 0.14, three widths away 0.01. So a Gaussian is local: beyond about three widths it has effectively vanished. But it never reaches exactly zero, which matters in Chapter 3.

Density. The bump divided by its area, φ(x) = g(x) / (σ√(2π)), so that it integrates to one. A narrow density is therefore tall and a wide one is low: halving σ doubles the peak.

The density is the version used when Gaussians compete with each other, so keep that last sentence in mind.

2.3 Two dimensions: distance becomes an ellipse

In the image plane a location is a pair x = (x₁, x₂) and the centre is a pair μ = (μ₁, μ₂). The simplest 2D Gaussian has its own width along each axis, s₁ and s₂. Distance in width units becomes

ρ² = ((x₁ − μ₁)/s₁)² + ((x₂ − μ₂)/s₂)²

and the bump is exp(−ρ²/2), exactly as in 1D; ρ (rho) is the distance measured in widths. All points with the same ρ have the same value, and those points form an ellipse with half-axes ρ·s₁ and ρ·s₂. So a 2D Gaussian looks like nested ellipses that fade outwards.

That ellipse is aligned with the axes. To tilt it we need a description that handles any orientation, and that is what a covariance matrix is.

Covariance matrix. A symmetric 2×2 matrix Σ (capital sigma) that says how spread out the Gaussian is in every direction. Distance in width units is ρ² = (x − μ)ᵀ Σ⁻¹ (x − μ), and the density is φ(x) = exp(−ρ²/2) / (2π √det Σ) Here Σ⁻¹ is the inverse matrix, det Σ its determinant, and ᵀ turns a column into a row.

For the axis-aligned case, Σ is diagonal with entries s₁² and s₂², and the formula gives back the ρ² above. A tilted Gaussian is the same thing turned through an angle θ: Σ = R diag(s₁², s₂²) Rᵀ, where R is the rotation by θ. Five numbers then describe the whole shape and position: two for the centre, two axis lengths and an angle.

2.4 Reading a covariance: a worked example

Take

Σ = 0.050.030.030.05

The diagonal entries are the variances along x₁ and x₂. The off-diagonal 0.03 is positive, so the two coordinates move together: the ellipse leans along the diagonal. To find its axes exactly, look for directions that Σ only stretches.

Eigenvector and eigenvalue. A direction v with Σv = λv: multiplying by Σ only scales it, by the number λ (lambda). For a covariance, the eigenvectors are the ellipse's principal axes, and each eigenvalue is the squared axis length.

Check by multiplying: Σ(1, 1) = (0.08, 0.08), so (1, 1) is an eigenvector with λ = 0.08. And Σ(1, −1) = (0.02, −0.02), so (1, −1) has λ = 0.02. The long axis runs along the diagonal with length √0.08 ≈ 0.283; the short axis is at right angles with length √0.02 ≈ 0.141.

Now see what this does to distance. Take two points both 0.2 from the centre:

  • Along the long axis: ρ² = 0.04/0.08 = 0.5, value exp(−0.25) = 0.78.
  • Along x₁: ρ² = 1.25 (worked out in the next step), value exp(−0.625) = 0.54.

Same distance on the page, different value. The covariance is a ruler that is longer in some directions than others.

2.5 Not every matrix is a covariance; the Cholesky factor fixes that

Suppose a network outputs the three entries of Σ directly and produces 0.05, 0.06, 0.05. Its eigenvalues are 0.11 and −0.01. A negative eigenvalue means a negative squared length: along that direction ρ² is negative, so exp(−ρ²/2) grows without limit. And det Σ is negative, so √det Σ does not exist. It is not a Gaussian at all.

Positive definite. A symmetric matrix whose eigenvalues are all positive; equivalently vᵀΣv > 0 for every non-zero v. For 2×2: the top-left entry and the determinant are both positive. Only such matrices are valid covariances.

Checking and repairing outputs during training is awkward. The clean fix is to store something from which Σ is built and which cannot go wrong.

Cholesky factor. A lower-triangular matrix L with positive diagonal, such that Σ = L Lᵀ. In 2D, L = a0bd, Σ = a²ababb² + d² with a > 0 and d > 0.

Two facts make this the right storage. Any a > 0, d > 0 and any b give a valid covariance (its determinant is (ad)² > 0). And every valid covariance has exactly one such factor. So nothing is lost and nothing can break.

For the example: a = √0.05 = 0.2236, b = 0.03/a = 0.1342, d = √(0.05 − b²) = 0.1789. Try the same on the bad matrix: d² = 0.05 − 0.072 < 0, so the factorisation itself refuses it.

The factor also makes the density cheap. For an offset (δ₁, δ₂) = x − μ, set u = δ₁/a and v = (δ₂ − b u)/d. Then ρ² = u² + v² and √det Σ = a·d; no matrix inverse is needed. For the point 0.2 along x₁: u = 0.894, v = −0.671, ρ² = 1.25, as promised.

Things to try: set the two axes equal and turn the angle (nothing changes, and b = 0); make axis 2 tiny (the peak density shoots up); turn the angle to 0 and 90 degrees (the off-diagonal entry vanishes). However you move the sliders, a and d stay positive.

2.6 Density value and mass

Put the pieces together. For one Gaussian the logarithm of its density at x is

log φ(x) = −log 2π − log a − log d − ½(u² + v²)

The first three terms are the height of the peak; the last says how far x is from the centre in width units. Narrow Gaussians (small a, d) have a tall peak and fall off fast. In the example the peak is 1/(2π · 0.04) ≈ 3.98.

One more number completes a Gaussian.

Mass (or weight). A positive number m that scales the whole density: the Gaussian contributes m·φ(x). In this project the masses of all Gaussians in an image are positive and add up to one.

On its own a mass only scales one bump up or down. It matters when several Gaussians share the image, which is Chapter 3.

2.7 This project's Gaussian: six numbers

Your model works in normalised image coordinates: the image is the square [−1, 1]², with x₂ pointing down so that row 0 is at the top (grid_points in geometry.py). The image is 2 units wide, so on a 32×32 grid one cell is 2/32 = 0.0625 wide.

The placement network outputs six unconstrained numbers per Gaussian. A decoder (_decode_free in free_gaussian_spatial.py) turns them into a valid Gaussian:

Raw outputsBecomesRuleGuarantee
2centre μtanh of eachinside the image, (−1, 1)²
2a, d0.008 + 0.05·softpluspositive, at least 0.008
1b0.05 × rawany value is allowed
1mass logitkept as ismasses = softmax over Gaussians

Softplus is a smooth function that is always positive (softplus(z) = log(1 + eᶻ)), and softmax turns a list of numbers into positive numbers that sum to one. So whatever the network outputs, the result is a valid Gaussian inside the image with masses that add up to one. With a raw output of zero, a = d ≈ 0.043, about two thirds of a 32×32 cell. The floor 0.008 is about 2 pixels at 512 pixels wide. (The later refinement stage, Part III, uses QDecode in free_gaussian_refine.py, with a = 0.004 + eʳ; same idea, a positive floor.)

The density itself is gaussian_log_density in geometry.py, and it is the formula of 2.6 line for line: zx = delta/a, zy = (delta − b·zx)/d, and log m − log 2π − log a − log d − ½(zx² + zy²). covariance_from_cholesky builds Σ exactly as in the definition box.

2.8 Check yourself

Answer from memory; each question gives its reason once you choose.

1. Why does the model store a Cholesky factor instead of the three entries of the covariance?

Both use three numbers, and the factor allows any tilt and elongation. Its point is that positive a and d always give a positive-definite Σ, so no network output can break the Gaussian.

2. For Σ with diagonal 0.05, 0.05 and off-diagonal 0.03, where is the long axis?

Σ(1, 1) = 0.08·(1, 1), the larger eigenvalue, and the axis length is its square root, 0.283. The direction (1, −1) has eigenvalue 0.02.

3. A network outputs covariance entries 0.05, 0.06, 0.05 directly. What is wrong?

Its eigenvalues are 0.11 and −0.01, and its determinant is negative. Along one direction the “distance” is negative and the bump grows without limit. The Cholesky factorisation fails on it, which is the point of storing the factor.

4. Two Gaussians share a centre; one has both axes 0.1, the other both axes 0.2. Compare their peak densities.

The peak is 1/(2π a d). Halving both axes divides a·d by four. Each density has total volume one, so the narrow one must be taller.

5. A reviewer asks what the six numbers per Gaussian are. Which answer is accurate?

Two numbers become the centre (through tanh), three the Cholesky factor (positive diagonal), one the mass logit (softmax across Gaussians). The feature vector is separate and comes in Chapter 4.

2.9 Close the loop

Without scrolling up, explain how six numbers describe one Gaussian in your model, and why they are not simply a centre and the three entries of Σ. Then compare with your first answer.

A model answer

“A 2D Gaussian is a centre and a covariance: the covariance is a symmetric 2×2 matrix whose eigenvectors are the ellipse's axes and whose eigenvalues are the squared axis lengths. It must be positive definite, or the density is not defined. Instead of the covariance, the model stores its Cholesky factor, a lower-triangular L with positive diagonal, because any such L gives a valid Σ = LLᵀ and the density is cheap to evaluate from it. The six numbers are the centre (two, kept inside [−1, 1]² by tanh), the factor (three, diagonal kept above a floor by softplus) and a mass logit (one; a softmax makes the masses positive and sum to one).”

Read next Bishop, “Pattern Recognition and Machine Learning” (2006), section 2.3, the opening pages on the geometry of the Gaussian (eigenvectors, elliptical contours, why the covariance must be positive definite). The free PDF is on that page.

Something unclear, or want to rehearse how you would say this to a reviewer? Ask your teacher in the chat.

One-page summary of this chapter ·