14.1 The hypothesis that did not survive
The research question asked for a model that places Gaussians “dividing hierarchically according to scene complexity”. Chapter 13 gave that idea a precise form: grow the set by splitting, score splits by error or by information gain, stop by description length, and let each image choose its own count. The project tested each piece before building it into the model. Each piece lost to something simpler.
These results changed the design: the project now uses one model that serves any count by direct clustering, not a split or merge tree. You will be asked about them, and they are among the strongest things you can show.
Before reading on: why do you think splitting lost to plain clustering? Write your best guess.
14.2 How to read these experiments
Every result below is a comparison at matched budget: the same number of Gaussians and the same total number of Adam refinement steps (Chapter 11) on both sides. Without that matching, a method that simply took more optimisation steps would look better for the wrong reason.
The baseline is always direct clustering: per-image k-means with Gaussians fitted to the clusters (Chapter 9), then the same refinement steps.
Caveats that apply to all of them. Write these on the slide, once.
- They are quick experiments in plain PyTorch, not the project's main training code, and no network was trained: no learned split or merge head was ever built.
- They use 50, 100 or 200 holdout images, not all 800, and one run each.
- Their settings (split offset, Adam step size 0.03, clustering width) were tuned on the same images they are scored on.
- The split study (s6) ran on the 16×16 target; merge-down and adaptive count (s7) on the 32×32 targets.
None of these caveats favours the baseline: tuning on the scored images flatters both arms, and the split arm had its offset and step size tuned as well.
14.3 Negative result 1: growing by splitting
What was done (s6, s6_split_hierarchy.json, 200 images, 16×16 target). Start from k-means with 16 Gaussians plus 8 Adam steps. Grow in rounds, 16 → 24 → 32 → 48 → 64 → 96. In each round, split the highest-scoring Gaussians into two children along the principal axis of their ellipse (Chapter 2): centres at ± 0.7 σ1, variance along that axis scaled by 1 − 0.7² so the second moment is preserved, mass halved. Then take 2 (or 4) Adam steps.
| At 64 Gaussians | 2 steps per round | 4 steps per round |
|---|---|---|
| Split by residual energy (best criterion) | 0.0000866 | 0.0000776 |
| Direct k-means, same steps | 0.0000622 | 0.0000585 |
| Split excess over k-means | +39% | +33% |
Splitting loses at every count tested, not only at 64.
Why it failed: a reasoned account.
- The best layouts are not nested. The best 64-Gaussian layout is not the best 32-Gaussian layout with each Gaussian cut in two. Clustering at 64 sees every token at once and puts each Gaussian on one group. A split child starts where its parent was, placed for a coarser budget, and a few Adam steps cannot move it far. Chapter 13's ordered-representation idea worked for the starting layout because a Halton prefix is spread evenly at every length; an adapted layout has no such property.
- A fresh split adds little to the least-squares basis. Under least squares (Chapter 4) overlapping Gaussians act as a smooth basis. Two children at ± 0.7 σ1 overlap heavily, so their columns of responsibilities are similar and the solve gains little until they move apart, which takes steps the budget does not provide.
- Greedy choices are myopic. Each round picks splits for their immediate benefit. The next step shows that even a perfect immediate score does not fix this, which points at the path, not the score.
Points 1 and 2 are interpretations consistent with the numbers, not separately measured. Point 3 is supported by the oracle result in 14.4.
14.4 Negative result 2: information gain is no better than random
Same procedure, four ways to choose which Gaussians to split (200 images, offset 0.7, 2 steps per round):
| Criterion, at 64 | 2 steps per round | 4 steps per round |
|---|---|---|
| Residual energy | 0.0000866 | 0.0000776 |
| Largest mass | 0.0000925 | 0.0000837 |
| Information gain | 0.0000937 | 0.0000845 |
| Random | 0.0000955 | 0.0000854 |
Information gain beats random by about 2% and loses to “split the heaviest Gaussian”. An oracle that actually tries each split, re-solves the least-squares features and measures the drop in error does no better than residual energy (50 images, at 64: 0.0000951 against 0.0000956 with 2 steps; 0.0000854 against 0.0000845 with 4).
Why information gain failed. It measures the wrong thing. The objective is the dense error after least squares. Information gain counts how many different prototype codes a Gaussian covers, and it is blind to how far apart those codes are: two nearly identical prototypes count as much as two very different ones, and a smooth gradient of features, which overlapping Gaussians fit well, can have high entropy. The adaptive-count runs below make this visible: counts chosen by information gain follow the image's code entropy (correlation 0.89) and barely follow its error (0.25).
Why even the oracle did not help. It measures the gain of one split immediately, before refinement moves anything. What matters is where the layout ends up after many rounds and steps; a split that looks best now can block a better arrangement later. This is the myopia of 14.3.
14.5 Negative result 3: merging down
If growing fails, try the reverse: cluster finely, then merge (s7, s7_fine_budget.json, first 100 holdout images, 32×32 targets). Start from 256 Gaussians (k-means + 16 steps). Repeatedly merge the pair whose merge is estimated to raise the error least, replacing it with one Gaussian that keeps the pair's total mass, mean and second moment; take 4 Adam steps after each halving. Compare with direct k-means at each count with the same total number of steps (20, 24, 28 and 32).
| Count | input512: merged | direct | excess | shift8 excess |
|---|---|---|---|---|
| 128 | 0.0000809 | 0.0000626 | +29% | +17% |
| 64 | 0.000135 | 0.000111 | +21% | +17% |
| 32 | 0.000232 | 0.000198 | +17% | +14% |
| 16 | 0.000365 | 0.000328 | +11% | +12% |
Why it failed. Two reasons, one measured.
- The merge score is a weak guide (measured). Checked against the true error increase on 10 images and 60 candidate pairs, the best cheap score (feature distance plus 0.25 times position distance, Ward-style) has a rank correlation of only 0.57 on
input512and 0.29 onshift8. Wrong pairs get merged. - Non-nesting again. The excess is largest straight after the first halving (+29% at 128), so this is not small errors piling up level by level. A merged layout is simply a worse 128-layout than one clustered for 128 directly: the same lesson as splitting, run backwards.
14.6 Negative result 4: a per-image count gains about 1%
The last idea keeps direct clustering and only lets the count vary by image (s7, 100 images). Each image gets 32, 48, 64, 96 or 128 Gaussians, with the mean held at 64, chosen from its own error curve.
| Allocation | input512 | shift8 |
|---|---|---|
| Oracle (reads the 16-step errors it is scored on) | −1.2% | −0.9% |
| Proxy (reads only the errors before refinement) | −1.0% | −0.5% |
Changes are against a fixed 64 for every image (input512: 0.0001154 oracle against 0.0001167 fixed). The oracle is an in-sample upper bound, and still gains about 1%. Under it, 74 of the 100 images kept 64.
On the split path in s6 (200 images, 16×16) an adaptive count did help splitting, 0.0000788 against 0.0000866 for fixed-count splits, and the count tracked the image's error closely (correlation 0.97 with residual energy). But matched-step k-means reached 0.0000596, so the adaptive split path still lost.
Why the gain is small. The count matters a great deal (Chapter 13: error roughly halves per doubling); its distribution across images does not. Moving Gaussians from one image to another helps only if images' error curves differ in shape, and here they are close to parallel on a log scale: between 32 and 128 Gaussians, each image's error falls by a factor of 2−0.85 per doubling, give or take 0.11 in the exponent. A back-of-envelope bound for this chapter from the same s7 errors: if every image followed its power law exactly and counts could take any value, the best reallocation would gain about 2.4%. With five allowed counts and real curves, 1% is most of what there was.
This is a statement about dense error on this data. A metric that weights some images or regions more, or a downstream task, could reward per-image counts differently. That is untested.
14.7 Why negative results strengthen the talk
A talk with only wins invites the question “what did you try that did not work?”, and a weak answer suggests the wins were found by search. Four reasons to show these results up front:
- They show the hypothesis was testable. The research question predicted that hierarchical, complexity-driven placement would help. It was tested at matched budget and it did not. That is how a claim earns trust.
- They pre-empt the obvious suggestion. “Why not grow the Gaussians by splitting?” is the first suggestion many listeners make; Chapter 15 shows where it comes from, and you have the numbers. Chapter 16 meets a published tokenizer that divides images by entropy; you have measured an entropy criterion against error.
- They explain a design choice. One model for any count by direct clustering is not an accident; it is what survived.
- They carry a transferable lesson. Good layouts at different budgets are not nested, and allocation across images matters far less than the total count.
How to present it: one slide, one matched comparison per row, the caveats in one line, and the consequence in one sentence. For example: “Splitting, merging and per-image counts all lost to direct clustering at equal count and equal refinement steps, by 11 to 39%, or gained about 1%. So the model clusters directly at whatever count you ask for.”
14.8 Check yourself
Answer from memory; each question gives its reason once you choose.
1. Why must the split arm and k-means use the same number of Adam steps?
2. The oracle split score, which measures each split's real gain, did no better than residual energy. What does that suggest?
3. Why does information gain choose splits poorly here?
4. A per-image count gains about 1% even with an oracle. The best explanation?
5. A reviewer says “your negative results are on quick experiments, so they prove nothing.” Your best reply?
14.9 Close the loop
Give the one-minute answer to “Your research question promised hierarchical placement. What happened?” Include one number per result and one reason.
A model answer
“We tested it before building it, always at equal count and equal refinement steps. Growing by moment-preserving splits lost to direct k-means by 39% at 64 Gaussians; the best criterion was residual energy, information gain was barely better than random, and even an oracle that measured each split's real gain did not help, so the problem is the greedy path: good layouts at different budgets are not nested. Merging down from 256 lost by 11 to 29%, partly because cheap merge scores rank pairs poorly (0.57 at best). And letting each image choose its count, at the same mean of 64, gained about 1% even with an oracle, because images' error curves are nearly parallel. These are quick experiments on 100 to 200 images with settings tuned on them, and no learned split network was trained. The consequence is the current design: one model that clusters directly at whatever count you ask for.”
Something unclear, or want to rehearse how you would say this to a reviewer? Ask your teacher in the chat.