Part I · Representing a signal · Chapter 3

Many Gaussians: summing, compositing and sharing

About 20 minutes, with one widget. Tap Continue to move on.

3.1 From one blob to an image

One Gaussian is a blob. An image needs many, and each will carry a value (later, a feature vector). At every location we must then decide how the blobs' values combine. There are three common rules, and they behave very differently. Your model uses the third, and that one choice explains why it behaves like a soft segmentation of the image.

By the end you can compare summation, compositing and normalised sharing; define a responsibility and say where the word comes from; and explain how your model turns Gaussians into a matrix A with one row per teacher token.

First, a guess: if two Gaussians carry the values 1 and 0.2, what value should a point halfway between them get? Write how you would decide.

3.2 Rule one: add them up

Give Gaussian k a value ck and a bump gk(x) with peak 1 (Chapter 2). The simplest rule adds the bumps, each scaled by its value:

f(x) = Σk ck gk(x)

A running example, on a line: Gaussian 1 at −0.5 with width 0.2 carries c₁ = 1; Gaussian 2 at 0.4 with width 0.3 carries c₂ = 0.2.

Location xg₁g₂Sum f(x)
−0.5 (centre of 1)1.0000.0111.002
0 (in between)0.0440.4110.126
0.9 (beyond 2)0.0000.2490.050

The result depends on how much Gaussian covers a point, not only which. At 0.9 the only Gaussian nearby carries 0.2, yet the field reads 0.05, because the bump has faded to a quarter. Where bumps overlap, values pile up. So to represent a flat region with summation you must tile it carefully with overlapping bumps whose sum is constant.

3.3 Rule two: stack them like films

Another rule treats each Gaussian as a semi-transparent film. Its opacity at x is αk(x) = ok gk(x), with ok between 0 and 1. Order the films front to back. The front film shows its value with weight α₁; the second shows through what the first lets pass, (1 − α₁); and so on:

f(x) = Σk ck αk(x) Πj<k (1 − αj(x))

Alpha compositing. Combining layers front to back, where each layer contributes its value times its opacity times the light not yet blocked by the layers in front.

In the example at x = 0, with Gaussian 2 in front and both opacities 1: 0.2 × 0.411 + 1 × 0.044 × (1 − 0.411) = 0.108. Swap the order and you get 1 × 0.044 + 0.2 × 0.411 × (1 − 0.044) = 0.122. The answer depends on the order.

In a 3D scene the order is natural: depth from the camera. In a single flat image there is no depth, so any order is arbitrary. And like summation, the result fades where no film is opaque. Chapter 15 shows where each rule is used in practice.

3.4 Rule three: share the point

The third rule asks a different question: which Gaussians own this point, and in what proportion? Each Gaussian makes a claim of size mk φk(x), its mass times its density (Chapter 2). The claims are divided by their total, so they become shares that add up to one.

Responsibility. The share of Gaussian k at location x, rk(x) = mk φk(x) / Σj mj φj(x) Every rk(x) is between 0 and 1, and at every x they sum to 1. The field is f(x) = Σk rk(x) ck, a weighted average of the Gaussians' values.

In the language of Chapter 2 this is a softmax over Gaussians of log mk + log φk(x), the log-density your code already computes. Same example, with masses 0.5 each:

xm₁φ₁m₂φ₂r₁, r₂SharedSum (3.2)
−0.50.9970.0070.993, 0.0070.9941.002
00.0440.2730.138, 0.8620.3100.126
0.90.0000.1660.000, 1.0000.2000.050

At 0.9 the shared field reads exactly 0.2, the value of the only Gaussian with a claim, however faint that claim is. Three consequences follow:

  • No gaps. Every point gets a full weighted average, even far from every centre.
  • No double counting. Overlap redistributes shares; it never adds values up.
  • No order. All Gaussians are treated alike.

3.5 Where the word “responsibility” comes from

The formula is not an invention of this project. It comes from a probability model.

Gaussian mixture model. A way to generate random points: pick Gaussian k with probability mk, then draw a point from it. The density of the resulting points is p(x) = Σk mk φk(x).

Now run it backwards. You see a point at x; which Gaussian produced it? By Bayes' rule, the probability that it was k is mk φk(x) / p(x), which is exactly rk(x). In the mixture-model literature this posterior is called the responsibility of component k for the point[Bishop, 2006]. Masses play the part of prior probabilities, which is why they are positive and sum to one.

Your model borrows the formula, not the probability story: nobody claims image locations are random draws. The word is still useful in a talk, because a statistician hears it and knows the exact meaning. Fitting a mixture to data is a separate topic; Chapter 9 covers it.

3.6 Width decides the shape of each region

With sharing, each Gaussian owns a soft region: the places where its responsibility is high. Together the regions cover the image, with smooth transitions between them.

Soft partition. A division of the image into overlapping regions given by weights that are non-negative and sum to one at every point. A hard partition is the special case in which every weight is 0 or 1.

Where do the boundaries fall? In the running example the 50/50 point is at x ≈ −0.11, not at the midpoint −0.05: the wider Gaussian reaches further. Three rules of thumb, all of which you can see in the widget:

  • Equal widths and masses: each point goes mostly to the nearest centre, and the boundary is halfway between centres.
  • Narrower Gaussians, sharper boundaries. As all widths shrink, the transitions approach a hard partition by nearest centre. Wider Gaussians give gentler blends.
  • Far away, the widest wins. A wide Gaussian's density falls more slowly, so it eventually out-claims a narrow one. A larger mass shifts boundaries outwards.

Things to try: make width 2 small and watch Gaussian 2's region shrink to a sliver while 1 and 3 take over; make width 1 large and see it claim the far edges; switch the plane to Plain sum and shrink the widths to see the gaps that sharing avoids.

3.7 Evaluating at the teacher's grid: the matrix A

The target your model must reproduce is a grid of feature vectors: one per cell, called a token, on a 16×16 or 32×32 grid (Part II explains where they come from). So the responsibilities are needed per cell, not at a single point.

Your code samples each cell at a 2×2 pattern of points, a quarter of a cell from its centre (on the 32×32 grid, ±0.0156). It averages each Gaussian's claim mkφk over the four points, and only then normalises over Gaussians. That is the mixture-model question asked for “a point somewhere in this cell” instead of one exact point. Averaging the responsibilities instead would give a different answer; the docstring of geometry.py says so explicitly.

The matrix A. Row p holds the responsibilities of all K Gaussians for token p. With P tokens it is P × K, every entry is between 0 and 1, and every row sums to 1. On the 32×32 grid with 64 Gaussians it is 1024 × 64.

A tiny version: a 1D “image” of four cells, centres −0.75, −0.25, 0.25, 0.75, with the two Gaussians of the running example.

TokenGaussian 1Gaussian 2If r were averaged
1 (−0.75)0.9980.0020.999, 0.001
2 (−0.25)0.8560.1440.758, 0.242
3 (0.25)0.0070.9930.009, 0.991
4 (0.75)0.0001.0000.000, 1.000

In the code: patch_quadrature places the points, patch_log_density averages the claims (a log-sum-exp minus log 4), and gaussian_responsibilities applies the softmax, all in geometry.py; on the GPU the same call runs a hand-written kernel (Chapter 18). FineGrid.points() in free_gaussian_fine.py supplies the points for the 32×32 grid. Once A is known, the placement has said everything it has to say. Chapter 4 decides what each Gaussian carries.

3.8 Check yourself

Answer from memory; each question gives its reason once you choose.

1. A point lies far from every Gaussian. What does the shared (normalised) field give there?

The shares are divided by their total, so they still sum to one. However faint the claims, the largest one dominates; far away that is usually the widest Gaussian. Summation, not sharing, fades to zero.

2. Why is alpha compositing an awkward fit for a single flat image?

Compositing weights each layer by what the layers in front let through, so the answer changes with the order. Depth gives the order in 3D; a single image has none.

3. Where does the word “responsibility” come from?

In a Gaussian mixture, rk(x) is the probability that point x was generated by component k, by Bayes' rule. The formula is identical.

4. All Gaussians are made much narrower, masses unchanged. What happens to the regions?

Narrow densities fall off fast, so the nearest one wins by a large factor and the transitions sharpen. Sharing never leaves gaps: the shares always sum to one.

5. A reviewer asks how a token's row of A is computed. Which answer is right?

The code averages mkφk over a 2×2 pattern inside the cell and then applies the softmax: the mixture's answer for a point somewhere in the cell. Averaging shares gives different numbers (0.758 against 0.856 in the example).

3.9 Close the loop

Without scrolling up, explain how your model combines many Gaussians at a location, why it uses that rule instead of summing or compositing, and what the matrix A is. Then compare with your first answer.

A model answer

“Each Gaussian claims a point with its mass times its density, and the claims are normalised to shares that sum to one: the responsibilities, the same quantity as the posterior in a Gaussian mixture model. The field is then a weighted average of the Gaussians' values. Unlike summation, it has no gaps and no double counting; unlike compositing, it needs no depth order. Each Gaussian owns a soft region whose sharpness is set by the widths. For the teacher's grid, the model averages each claim over a 2×2 pattern inside each cell and then normalises, giving a matrix A of tokens by Gaussians whose rows sum to one.”

Read next Bishop, “Pattern Recognition and Machine Learning” (2006), the opening of section 9.2, “Mixtures of Gaussians”, up to the definition of responsibilities. Stop before the EM algorithm; that is Chapter 9.

Something unclear, or want to rehearse how you would say this to a reviewer? Ask your teacher in the chat.

One-page summary of this chapter ·