Part VI · Doing it properly · Chapter 19

Evaluation and rigour

About 25 minutes. Tap Continue to move on; nothing later depends on anything you have not yet seen.

19.1 Where a careful reviewer attacks

A reviewer who cannot fault the idea will fault the evidence. Which metric? On which images? How many seeds? Were the settings chosen on the test images? Against which baseline, at which budget? Every number in this book so far came with caveats in brackets. This chapter explains why each caveat exists, what it costs, and how to say it without weakening the talk.

By the end you can explain what each of the project's metrics can and cannot show, why the Pet mIoU gain was withdrawn, how to read seed spread against effect size, and you can recite the project's threats to validity before a reviewer does.

Before reading on: list the weaknesses in the project's evaluation that you already know about.

19.2 The primary metric: dense error on held-out images

Holdout dense MSE. The mean squared difference between the Gaussian field and the teacher's feature map, over all tokens and all 384 dimensions, averaged over images the model was not trained on (here 800 holdout images; training uses 6,400).

It is the right metric for the claim “the Gaussians represent the teacher's field”: it is dense, it sees every token, and it moves with placement. Gridless64 against Anchored64 (the baseline whose 64 Gaussians stay anchored to a grid) differs by 11 to 12% in it, while the classification probe below cannot tell them apart.

Three things to say about it, each with a project number.

  • It is fidelity, not semantics. Low error means close to DINOv3-S features. Whether that helps a task is a separate question (19.3, and roadmap P1).
  • Which images matters. Trained models are scored on 800 images; the quick experiments on the first 100 or 200. Model J at 64 Gaussians on input512 scores 0.0001137 on all 800 and 0.0001119 on the first 100, a 1.6% difference from the cohort alone. Compare like with like, and say which cohort a number comes from.
  • The “optimum” is itself an estimate. The per-image optimum (Chapter 12) is Adam run for a fixed number of steps from a chosen start. On the 16×16 target, 600 steps from a grid give 0.0000465 and from the Gridless64 layout 0.0000558. It is a reference point, not a proven lower bound.

19.3 Secondary metrics: a probe that cannot see placement, and a gain that did not replicate

Linear probes. A linear classifier trained on frozen features reads how linearly accessible a property is; fair practice tunes its regularisation and compares methods under the same protocol[Kornblith et al., 2018]. The project's 100-class probe on the 800 holdout images gives about 0.87 top-1 (0.866 to 0.878) for every 16×16 arm, from the weighted-mean model to refinement J, while dense error fell by more than half. Between two arms of the same seed the difference is one or two images in 800.

The likely reason it is blind here: the probe reads a pooled summary of the field. Least squares reproduces the image's average feature well whatever the layout, so placement, which changes where features sit, barely changes the summary. That is a property of the probe, not evidence that placement is useless. On input512 the probe reaches 0.889, against 0.921 for the teacher's class token and 0.866 for the mean of its patch tokens.

The Pet segmentation gain. On Oxford-IIIT Pet foreground segmentation[Parkhi et al., 2012], 148 images, Gridless64 minus Anchored64 in image-averaged mIoU was:

Training seedDifference95% interval over images
20261025+0.0031[+0.0002, +0.0062]
20261108+0.0039[+0.0009, +0.0069]
20261112−0.0002[−0.0023, +0.0021]

Two intervals excluded zero; the third seed erased the effect. The lesson is precise: those intervals were a bootstrap over images, so they measured image-to-image variation for one trained model. They said nothing about variation between training runs, which turned out to be as large as the effect. The scores also sit near a ceiling (image-averaged mIoU 0.981 to 0.985; the teacher's global mIoU is 0.990), where small differences are fragile. Present it as “two of three seeds; not robust”, or leave it out.

19.4 Seeds and intervals: read spread against effect

Training runs differ by seed, and so can conclusions; reinforcement learning showed how often single runs mislead[Henderson et al., 2017]. Seeds are not the only source of variance either: data order, splits and hyperparameter choice all add to it[Bouthillier et al., 2021].

The useful habit is to compare the effect with the seed spread. The project's three-seed results, holdout dense MSE:

ArmSeedsSpreadEffect it supports
Least-squares Gridless640.0001041, 0.0001050, 0.0001042under 1%12 to 13% below Anchored64 in every seed
Clustering B0.0000804, 0.0000806, 0.00008050.24% of the mean23% below least-squares Gridless64
Refinement J0.0000587, 0.0000588, 0.0000589about 0.3%27% below clustering B

When the effect is ten to a hundred times the spread, as here, one seed was enough to make the decision, which is why development uses one seed per arm. It is not enough to report the number: final numbers need several seeds with intervals over seeds and over images.

The same habit tells you what you cannot claim. Removing the coverage penalty changed the error by +0.10%: below anything resolved, so the honest reading is “no detectable effect”, not “slightly worse”.

With few runs, report interval estimates rather than single means, for instance by a stratified bootstrap, and prefer robust summaries[Agarwal et al., 2021]. When two methods are scored on the same images, use a paired test[Dror et al., 2018].

19.5 Tuning on the evaluation images, and the sealed test set

Every choice made by looking at a set of images makes that set a little less independent. In this project the 800-image holdout has been reused throughout, in three ways:

  1. Checkpoint selection. The runner saves best.pt by holdout dense MSE. In the refinement arms the best checkpoint is up to about 2% better than the final one (H: best 0.0000605 at step 1500, final 0.0000619; I: 0.0000562 against 0.0000571; for J they coincide).
  2. Design decisions. Which arm to keep, which target (input512), how much deep-supervision weight: all decided on holdout numbers.
  3. Quick-experiment settings. Clustering width, split offset, step sizes and the oracle allocations were tuned or computed on the very images they are scored on.
Sealed test set. Images set aside before any decision, never looked at during development, and scored once at the end for the final numbers. The roadmap plans 2,000 images disjoint from the 6,400/800 cohort; it has not been created yet.

How bad is the reuse? For effects of 20% with 0.3% seed spread, selection bias of a few percent cannot reverse a ranking. For small differences, such as 4% between trained and untrained refinement, or 1% for adaptive count, it could. The sealed set settles this; until it exists, say exactly what the holdout was used for.

Venues now ask for this in writing: the NeurIPS checklist asks about claims, limitations, statistical significance and compute[NeurIPS, 2026].

19.6 Baselines at equal budget, and the downsampling caution

A compression method must beat the simplest compressor at the same budget. A recent study of visual-token compression for multimodal language models found that simple image downsampling outperforms many advanced compression methods across widely used benchmarks[Liao et al., 2025]. The lesson transfers: the obvious baseline is often the hardest.

Where the project stands, on the 16×16 target at 64 tokens (the static layouts are quick experiments; all rows 800 holdout images):

Method at 64Dense MSE
Static 8×8 grid of wide Gaussians (close to smooth pooling)0.000134
Least-squares Gridless64, trained0.000104
Per-image k-means, no refinement0.000086
Clustering B, trained, one pass0.0000804
Refinement J, trained, 16 steps0.0000587

Note the uncomfortable row: the trained one-shot predictor was worse than per-image k-means. Only the clustering stage beat it. That is why per-image k-means, with the same refinement steps, is the reference in the budget curve of Chapter 13 and in every negative result of Chapter 14.

What is missing: the formal phase-1 baselines (average pooling, token merging, per-image k-means as tokens) run through the same evaluation at 16 to 256; a downstream task at matched token count; and the arms of earlier comparisons matched in capacity and compute (Gridless64 and Anchored64 differ in both, which the run notes say).

19.7 Threats to validity for this project

Learn this list. A reviewer who hears it from you first has much less to attack.

Internal: could something other than the method explain the numbers?

  1. The 800-image holdout has been used for checkpoint selection and every design decision; no sealed test set exists yet.
  2. Quick-experiment settings and oracle allocations were tuned or computed on the images they are scored on.
  3. Most arms have one seed; three seeds exist only for least-squares Gridless64 and Anchored64, clustering B and refinement J on the 16×16 target. Everything on 32×32, including the any-count curve, is one seed.
  4. Quick experiments run in plain PyTorch with slightly different settings (for example a relative ridge of 1e-3 rather than λ = 0.01), not the Triton training path.
  5. Gridless64 and Anchored64 differ in capacity and compute, so their gap is not purely placement.

Construct: does the metric measure the claim?

  1. Dense MSE measures fidelity to DINOv3-S features, not semantic usefulness; the probe is insensitive to placement; the one dense task gain (Pet) did not replicate.
  2. The 16×16 target saturates above about 96 Gaussians; the 32×32 input512 target comes from 512 px input, so it is a different field, not a sharper view of the same one.
  3. The per-image optimum is a finite Adam run whose value depends on its start (0.0000465 against 0.0000558), so gaps “to the optimum” are approximate.

External: does it generalise?

  1. One cohort (6,400 training, 800 holdout images, 100 classes), one teacher (DINOv3 ViT-S/16), one GPU; no standard dense benchmark (ADE20K, VOC, NYUv2) yet.
  2. The teacher is needed at inference; the representation compresses features already computed, so its value depends on the application (roadmap P4).

Conclusion: are the inferences sound?

  1. Intervals so far are over images, never over seeds; differences of about 1% (coverage penalty, adaptive count) are below what has been resolved.
  2. Comparisons across cohorts (100 against 800 images) mix a 1 to 2% cohort effect into the gap.
  3. The comparison between the any-count model and models trained at each count is one seed on final checkpoints; at 128 and 256 the per-count runs drifted after early updates, so a different stopping point would change the gap.
  4. The closest work (GViT v2, GPSToken) has not been read in full, so any novelty statement is limited to a search.

19.8 Check yourself

Answer from memory; each question gives its reason once you choose.

1. The Pet mIoU interval excluded zero in two seeds, then the third seed showed no gain. What did the intervals fail to capture?

They were bootstrapped over images for one trained model. Run-to-run variation was as large as the effect, so only seeds could reveal it.

2. Why does the classification probe not separate the placement arms?

Least squares reproduces the average feature well under any layout; placement moves features within the image, which pooling discards. Dense error does see it.

3. Clustering B's three seeds spread by 0.24%; it is 23% below the previous model. What follows?

An effect a hundred times the spread justifies the development decision. Reported numbers still need intervals over seeds and a sealed test set.

4. Which result is most at risk from selecting checkpoints on the holdout?

Best-checkpoint selection moved refinement arms by up to about 2%, comparable to a 4% effect; it cannot reverse 12% or 23%.

5. Why does the downsampling result matter to this project?

It is a caution, not a verdict on Gaussians: always include the plain baseline at the same budget. Here the trained one-shot predictor lost to per-image k-means.

19.9 Close the loop

A reviewer asks: “How much should I trust your numbers?” Answer in under a minute, naming what is solid and the main threats.

A model answer

“The main effects are solid: least squares, clustering and refinement each lower holdout dense error by 22 to 27%, with three-seed spreads under 1%. What is weaker: the holdout has been used for every decision and for checkpoint selection, quick experiments were tuned on the images they score, most 32×32 results are one seed, and there is no sealed test set yet. Dense error measures fidelity to the teacher, not semantics; the classification probe cannot see placement, and an early Pet segmentation gain did not survive a third seed, so I no longer claim it. External validity is one cohort, one teacher, and the teacher is needed at inference. Final numbers will come from at least three seeds on a sealed 2,000-image set, with intervals over seeds and images, beside pooling and k-means at the same budget.”

Read next Agarwal et al., “Deep Reinforcement Learning at the Edge of the Statistical Precipice” (2021), sections 1 to 4. Written for reinforcement learning, but its argument and tools (interval estimates and stratified bootstrap with few runs) apply directly to reporting a handful of seeds.

Something unclear, or want to rehearse how you would say this to a reviewer? Ask your teacher in the chat.

One-page summary of this chapter ·