3.1 From one blob to an image
One Gaussian is a blob. An image needs many, and each will carry a value (later, a feature vector). At every location we must then decide how the blobs' values combine. There are three common rules, and they behave very differently. Your model uses the third, and that one choice explains why it behaves like a soft segmentation of the image.
First, a guess: if two Gaussians carry the values 1 and 0.2, what value should a point halfway between them get? Write how you would decide.
3.2 Rule one: add them up
Give Gaussian k a value ck and a bump gk(x) with peak 1 (Chapter 2). The simplest rule adds the bumps, each scaled by its value:
f(x) = Σk ck gk(x)
A running example, on a line: Gaussian 1 at −0.5 with width 0.2 carries c₁ = 1; Gaussian 2 at 0.4 with width 0.3 carries c₂ = 0.2.
| Location x | g₁ | g₂ | Sum f(x) |
|---|---|---|---|
| −0.5 (centre of 1) | 1.000 | 0.011 | 1.002 |
| 0 (in between) | 0.044 | 0.411 | 0.126 |
| 0.9 (beyond 2) | 0.000 | 0.249 | 0.050 |
The result depends on how much Gaussian covers a point, not only which. At 0.9 the only Gaussian nearby carries 0.2, yet the field reads 0.05, because the bump has faded to a quarter. Where bumps overlap, values pile up. So to represent a flat region with summation you must tile it carefully with overlapping bumps whose sum is constant.
3.3 Rule two: stack them like films
Another rule treats each Gaussian as a semi-transparent film. Its opacity at x is αk(x) = ok gk(x), with ok between 0 and 1. Order the films front to back. The front film shows its value with weight α₁; the second shows through what the first lets pass, (1 − α₁); and so on:
f(x) = Σk ck αk(x) Πj<k (1 − αj(x))
In the example at x = 0, with Gaussian 2 in front and both opacities 1: 0.2 × 0.411 + 1 × 0.044 × (1 − 0.411) = 0.108. Swap the order and you get 1 × 0.044 + 0.2 × 0.411 × (1 − 0.044) = 0.122. The answer depends on the order.
In a 3D scene the order is natural: depth from the camera. In a single flat image there is no depth, so any order is arbitrary. And like summation, the result fades where no film is opaque. Chapter 15 shows where each rule is used in practice.
3.4 Rule three: share the point
The third rule asks a different question: which Gaussians own this point, and in what proportion? Each Gaussian makes a claim of size mk φk(x), its mass times its density (Chapter 2). The claims are divided by their total, so they become shares that add up to one.
In the language of Chapter 2 this is a softmax over Gaussians of log mk + log φk(x), the log-density your code already computes. Same example, with masses 0.5 each:
| x | m₁φ₁ | m₂φ₂ | r₁, r₂ | Shared | Sum (3.2) |
|---|---|---|---|---|---|
| −0.5 | 0.997 | 0.007 | 0.993, 0.007 | 0.994 | 1.002 |
| 0 | 0.044 | 0.273 | 0.138, 0.862 | 0.310 | 0.126 |
| 0.9 | 0.000 | 0.166 | 0.000, 1.000 | 0.200 | 0.050 |
At 0.9 the shared field reads exactly 0.2, the value of the only Gaussian with a claim, however faint that claim is. Three consequences follow:
- No gaps. Every point gets a full weighted average, even far from every centre.
- No double counting. Overlap redistributes shares; it never adds values up.
- No order. All Gaussians are treated alike.
3.5 Where the word “responsibility” comes from
The formula is not an invention of this project. It comes from a probability model.
Now run it backwards. You see a point at x; which Gaussian produced it? By Bayes' rule, the probability that it was k is mk φk(x) / p(x), which is exactly rk(x). In the mixture-model literature this posterior is called the responsibility of component k for the point[Bishop, 2006]. Masses play the part of prior probabilities, which is why they are positive and sum to one.
Your model borrows the formula, not the probability story: nobody claims image locations are random draws. The word is still useful in a talk, because a statistician hears it and knows the exact meaning. Fitting a mixture to data is a separate topic; Chapter 9 covers it.
3.6 Width decides the shape of each region
With sharing, each Gaussian owns a soft region: the places where its responsibility is high. Together the regions cover the image, with smooth transitions between them.
Where do the boundaries fall? In the running example the 50/50 point is at x ≈ −0.11, not at the midpoint −0.05: the wider Gaussian reaches further. Three rules of thumb, all of which you can see in the widget:
- Equal widths and masses: each point goes mostly to the nearest centre, and the boundary is halfway between centres.
- Narrower Gaussians, sharper boundaries. As all widths shrink, the transitions approach a hard partition by nearest centre. Wider Gaussians give gentler blends.
- Far away, the widest wins. A wide Gaussian's density falls more slowly, so it eventually out-claims a narrow one. A larger mass shifts boundaries outwards.
Things to try: make width 2 small and watch Gaussian 2's region shrink to a sliver while 1 and 3 take over; make width 1 large and see it claim the far edges; switch the plane to Plain sum and shrink the widths to see the gaps that sharing avoids.
3.7 Evaluating at the teacher's grid: the matrix A
The target your model must reproduce is a grid of feature vectors: one per cell, called a token, on a 16×16 or 32×32 grid (Part II explains where they come from). So the responsibilities are needed per cell, not at a single point.
Your code samples each cell at a 2×2 pattern of points, a quarter of a cell from its centre (on the 32×32 grid, ±0.0156). It averages each Gaussian's claim mkφk over the four points, and only then normalises over Gaussians. That is the mixture-model question asked for “a point somewhere in this cell” instead of one exact point. Averaging the responsibilities instead would give a different answer; the docstring of geometry.py says so explicitly.
A tiny version: a 1D “image” of four cells, centres −0.75, −0.25, 0.25, 0.75, with the two Gaussians of the running example.
| Token | Gaussian 1 | Gaussian 2 | If r were averaged |
|---|---|---|---|
| 1 (−0.75) | 0.998 | 0.002 | 0.999, 0.001 |
| 2 (−0.25) | 0.856 | 0.144 | 0.758, 0.242 |
| 3 (0.25) | 0.007 | 0.993 | 0.009, 0.991 |
| 4 (0.75) | 0.000 | 1.000 | 0.000, 1.000 |
In the code: patch_quadrature places the points, patch_log_density averages the claims (a log-sum-exp minus log 4), and gaussian_responsibilities applies the softmax, all in geometry.py; on the GPU the same call runs a hand-written kernel (Chapter 18). FineGrid.points() in free_gaussian_fine.py supplies the points for the 32×32 grid. Once A is known, the placement has said everything it has to say. Chapter 4 decides what each Gaussian carries.
3.8 Check yourself
Answer from memory; each question gives its reason once you choose.
1. A point lies far from every Gaussian. What does the shared (normalised) field give there?
2. Why is alpha compositing an awkward fit for a single flat image?
3. Where does the word “responsibility” come from?
4. All Gaussians are made much narrower, masses unchanged. What happens to the regions?
5. A reviewer asks how a token's row of A is computed. Which answer is right?
3.9 Close the loop
Without scrolling up, explain how your model combines many Gaussians at a location, why it uses that rule instead of summing or compositing, and what the matrix A is. Then compare with your first answer.
A model answer
“Each Gaussian claims a point with its mass times its density, and the claims are normalised to shares that sum to one: the responsibilities, the same quantity as the posterior in a Gaussian mixture model. The field is then a weighted average of the Gaussians' values. Unlike summation, it has no gaps and no double counting; unlike compositing, it needs no depth order. Each Gaussian owns a soft region whose sharpness is set by the widths. For the teacher's grid, the model averages each claim over a 2×2 pattern inside each cell and then normalises, giving a matrix A of tokens by Gaussians whose rows sum to one.”
Something unclear, or want to rehearse how you would say this to a reviewer? Ask your teacher in the chat.