Part I · Representing a signal · Chapter 5

Putting it together: is this a neural representation?

About 20 minutes. Defence question: “Is this an implicit neural representation at all?”

5.1 The question you will be asked

Your research question calls the model a “semantically relevant, semi-implicit neural representation”. Each of those words will be tested. You now have every piece of the read-out: Gaussians (Chapter 2), responsibilities (Chapter 3) and least-squares features (Chapter 4). This chapter assembles them, says how the result is scored, and answers the question with Chapter 1's two questions, including the parts of the answer that are uncomfortable.

By the end you can write the whole read-out in one line, state what dense MSE measures, place the model with Chapter 1's two questions, and give a one-sentence answer to “is it a neural representation?” that survives a hostile follow-up.

Write your current one-sentence answer to the defence question. You will compare it at the end.

5.2 The whole read-out

For an image with K Gaussians, each with centre μk, Cholesky factor Lk, mass mk and feature zk, the value at any location x is

F̂(x) = Σk rk(x) zk, rk(x) = mkφk(x) / Σj mjφj(x)

Evaluated on the teacher's grid this is F̂ = AZ. The order of work for one image:

Teacher tokens T (32×32 × 384) Placement network (Part III) 6 numbers per Gaussian (Ch. 2) Responsibilities A (Ch. 3) Ridge solve for Z (Ch. 4) Field AZ → dense MSE against T T is used again here

In code this is FineRefineModel.forward in free_gaussian_fine.py: the decoder gives centres, factors and mass logits; last_axis_softmax turns logits into masses; gaussian_responsibilities gives A; final_coefficients_any solves for Z; and runtime_matmul(weights, semantic) forms the field.

How compact is it? With 64 Gaussians the representation is 64 × (6 + 384) = 24,960 numbers. The 32×32 teacher grid holds 1024 × 384 = 393,216, about 16 times more. Almost all of the 24,960 are features; the geometry is only 384 numbers.

5.3 How it is scored: dense MSE

Dense MSE. The mean, over every token and every channel, of the squared difference between the field and the teacher: (1/PD) Σp,d (F̂pd − Tpd)², computed on held-out images.

The numbers look tiny, so it helps to calibrate them. Each teacher row has length one, so its 384 squared entries add up to one and average 1/384 ≈ 0.0026. Multiplying a dense MSE by 384 therefore gives the squared error per token as a fraction of the token's squared length.

Model, 64 GaussiansTargetDense MSE× 384
Gridless64, least squares (Chapter 4)16×160.0001041about 4.0%
Current model16×160.0000587about 2.3%
Current model32×32 (512 px input)0.0001137about 4.4%

All rows: 800 holdout images, one seed, the holdout reused during development. The two targets are different fields, so the 16×16 and 32×32 rows describe how hard each target is, not a gain or loss; Chapter 8 explains why the finer one is harder.

Why this metric? It is exactly the quantity the least-squares solve minimises, so features and score agree. What it does not show is whether the field keeps the meaning of the features; that is tested separately with probes (Chapter 19).

5.4 Chapter 1's two questions, applied

How is the signal read at a location? By a fixed rule over explicit parts: compute the responsibilities and average the features. No network is evaluated at x, and the rule is defined at every location, not only on the grid.

How are the stored parameters found? In two different ways, and the difference matters:

PartFound byFamily trait
Placement: centre, factor, massOne pass of a network trained on 6,400 imagesAmortised
Features zkA closed-form ridge solve against this image's teacher tokensPer image, but exact and non-iterative
Read-outResponsibility-weighted averageFixed rule, defined at any x
PartsEach Gaussian has a place, a size, a soft region and a featureExplicit, interpretable

Compare Chapter 1's examples. NeRF and SIREN read the signal by evaluating a network at x, and they optimise each signal from scratch[Mildenhall et al., 2020][Sitzmann et al., 2020]. Your model reads with a fixed rule and finds its placement in one forward pass. By Chapter 1's definitions it is a hybrid representation with amortised placement.

One precise caveat. The formula is continuous, but every reported number evaluates it on the teacher's grid. Whether values between tokens are meaningful has not been measured; the roadmap lists a resolution-free query test (P3) that is still open. Say “defined anywhere”, not “accurate anywhere”.

5.5 The answer, and the word “semi-implicit”

So, is it a neural representation? Not in the narrow sense of a neural field, because no network is evaluated at a location. In the broader sense, yes: it is a continuous representation whose parameters are produced by a neural network. A defensible one-sentence definition:

What the model is. An explicit, finite set of continuous primitives, read out by a fixed normalised-mixture rule, whose placement is predicted by an amortised network and whose features are solved in closed form.

That is what the roadmap's “semi-implicit” is reaching for: continuous like an implicit field, explicit like a grid. The neural-fields survey calls this middle ground hybrid representations[Xie et al., 2022]; no source was found that defines “semi-implicit representation” formally.

Two cautions before you use the word in a talk:

  • The term is taken. “Semi-implicit” already names a family of variational inference methods, semi-implicit variational inference[Yin and Zhou, 2018]. A reviewer from probabilistic machine learning may hear that meaning. Define your use on the first slide, or say “hybrid explicit–implicit feature field”.
  • “Neural” is earned by placement only. The features come from a solve, not from the network. That leads straight to the hardest question.

5.6 The honest caveat: the teacher at inference

Look again at the diagram: T is used twice. The placement network reads the teacher's tokens, and the ridge solve needs them to compute Z. So to represent a new image you must first run the teacher on it. The Gaussians are a compact, structured re-description of the teacher's field, not a replacement for the teacher. The roadmap lists this among the known weaknesses (“the teacher is in the inference path”) and plans to address it as P4: either a pixel encoder that predicts Gaussians and features directly, or a clear statement that the method is a tokeniser for foundation-model features, with an application where that pays.

When the question comes, do three things:

  1. Concede. “Yes. Today the features are the closed-form optimum given placement, solved against the teacher.”
  2. Scope. “The contribution is placement: where to put a small number of Gaussians so that the best possible features reconstruct the field well. The solve makes that question clean, because features are always optimal for the placement.”
  3. Point forward. “Removing the teacher from inference is the next step on the roadmap, P4.”

Do not claim the opposite. “The teacher is only used during training” is false for this model, and a reviewer who has read the method will know.

5.7 Check yourself

Answer from memory; each question gives its reason once you choose.

1. Which property makes the model resemble an implicit neural field?

Being defined at every x is the field-like property. Being produced by a network is amortisation, a separate question; one feature per token is the explicit grid it replaces.

2. What does a dense MSE of 0.0001137 mean on unit-length 384-channel targets?

The mean is over 384 channels whose squares sum to one per token, so multiply by 384: 0.0001137 × 384 ≈ 0.044.

3. How are the two kinds of parameter found?

A trained network outputs the six numbers per Gaussian in one pass (amortised); the features are the ridge least-squares solution against the teacher tokens of that image.

4. Why should you define “semi-implicit” on your first slide?

Semi-implicit variational inference (Yin and Zhou, 2018) uses the term with a different meaning. Defining yours, or saying “hybrid explicit–implicit”, avoids the confusion.

5. A reviewer says the features need the teacher at test time, so this is not a representation. Best reply?

Both other replies are false for the current model. Concede, scope the claim to placement, and point to roadmap item P4 on teacher-free features.

6. Is it safe to say the field is accurate between the teacher's tokens?

Every reported error compares against the teacher's grid. The sample points average claims inside a cell; they do not test values against a finer target. The resolution-free test (P3) is still open.

5.8 Close the loop

Without scrolling up, answer “is this an implicit neural representation?” in two or three sentences, using the words continuous, explicit, amortised and teacher. Then compare with your first answer.

A model answer

“It is a hybrid. Like a neural field it is continuous and defined at any location, but its parameters are an explicit, interpretable set of Gaussians read out by a fixed normalised-mixture rule, not a network evaluated at each point. Placement is amortised by a network; the features are the closed-form ridge least-squares optimum given that placement, which today means the teacher is needed at inference, so the contribution is the placement, and teacher-free features are the next step.”

Read next Xie et al., “Neural Fields in Visual Computing and Beyond” (2022), the part on hybrid representations. It is the closest thing to a formal home for “semi-implicit” and gives you the field's own words for the answer above.

Something unclear, or want to rehearse how you would say this to a reviewer? Ask your teacher in the chat.

One-page summary of this chapter ·