Summary · Chapter 19

Evaluation and rigour

Holdout dense MSE: fidelity to the teacher field on 800 unseen images. Sensitive to placement; not a semantic measure. Say the cohort: J at 64 on input512 is 0.0001137 on 800 images, 0.0001119 on the first 100. The per-image optimum is a finite Adam run (0.0000465 from a grid, 0.0000558 from Gridless64).

Linear probe: about 0.87 top-1 for every 16×16 arm; it reads a pooled summary that placement barely changes (probing practice).

Pet mIoU: +0.0031 and +0.0039 with image-bootstrap intervals excluding zero, then −0.0002 [−0.0023, +0.0021] on a third seed. Intervals over images miss run-to-run variation. “Two of three seeds; not robust.”

Seeds: compare effect with spread. Three-seed spreads under 1% against effects of 12 to 27%: safe decisions, but report with intervals over seeds (Agarwal et al.). A +0.10% change is “no detectable effect”.

Holdout reuse: checkpoint selection (best up to about 2% below final), design decisions, tuned quick-experiment settings. Remedy: a sealed 2,000-image test set, not yet created.

Baselines: the simplest compressor at equal budget is often hardest (downsampling result). Trained one-shot Gridless64 (0.000104) lost to per-image k-means (0.000086); clustering B (0.0000804) and J (0.0000587) beat it.

Threats to validity

  1. Holdout used for selection and decisions; no sealed test set.
  2. Quick experiments tuned and oracles computed in-sample.
  3. One seed for most arms and for everything on 32×32.
  4. Quick experiments in plain PyTorch with slightly different settings.
  5. Gridless64 and Anchored64 differ in capacity and compute.
  6. Dense MSE is fidelity, not semantics; probe blind; Pet gain withdrawn.
  7. 16×16 target saturates; input512 is a different field.
  8. The per-image optimum depends on its start.
  9. One cohort, one teacher, one GPU; no standard dense benchmarks.
  10. Teacher needed at inference.
  11. No intervals over seeds; 1% differences unresolved.
  12. Cross-cohort comparisons (100 vs 800 images).
  13. Per-count comparison: one seed, final checkpoints, drift at 128 and 256.
  14. Closest work not read in full.