Part I · Representing a signal · Chapter 1

Signals and how to store them

About 18 minutes. Tap Continue to move on; nothing later depends on anything you have not yet seen.

1.1 The question this book answers

Your research asks whether a small set of Gaussians that carry semantic information can serve as a neural representation of an image. Before we can judge that, we need to agree on what a representation is and what kinds exist. That is all this chapter does.

By the end you can sort any way of storing an image into one of three families, and ask the two questions that tell representations apart.

First, write down what you think a “representation” of an image is. One sentence is enough; we come back to it at the end.

1.2 A signal is a function of location

Think of an image not as a table of numbers but as a rule: give it a location x in the image plane, and it gives back a value.

Signal. A function f : Ω → ℝD from locations Ω (here, points in the image) to a list of D numbers.
  • A greyscale image: D = 1, a brightness.
  • A colour image: D = 3, red, green and blue.
  • A feature map: D in the hundreds. Each location carries a long list of numbers that a network computed. In your work D = 384. Where these numbers come from and what they mean is Part II; for now, treat them as “many numbers per location”.

The signal itself is ideal and continuous. Every way of storing it is a compromise, and the compromise chosen is what we call the representation.

1.3 Family one: store the samples

The obvious approach is to measure the signal on a regular grid and keep the measurements. A photo is a grid of pixels. The feature map your work starts from is a grid too: 16 by 16 cells, later 32 by 32, each holding 384 numbers.

Explicit representation. The stored numbers are the signal's values at chosen locations; reading the signal is a lookup.

Two consequences follow directly:

  • Cost grows with resolution. Doubling the grid's width and height stores four times as many values. Your 16×16 grid holds 16 × 16 × 384 = 98,304 numbers; the 32×32 grid holds 393,216.
  • The signal only exists at the grid points. To read it anywhere else you must invent a rule, such as copying the nearest cell or blending the four nearest (bilinear interpolation). The grid itself says nothing about what lies between its points.

Keep the second point in mind. A 16×16 feature grid over a whole image is very coarse: on the 256-pixel images your work starts from, every cell covers 16 by 16 pixels. Chapter 8 shows how much this hid in your data.

1.4 Family two: store a function

Instead of samples, store a small neural network fθ that takes a location and returns the value there. The network's weights θ are the stored image. To create them you train the network on that one image until its outputs match the pixels.

Implicit representation (also neural field). The signal is stored as the parameters of a function that can be evaluated at any location.

Two well-known examples:

  • NeRF stores a whole 3D scene as one network mapping a 3D position and viewing direction to colour and density[Mildenhall et al., 2020].
  • SIREN shows that networks with sine activations can store images and other signals with fine detail[Sitzmann et al., 2020].

What you gain: the signal is defined everywhere, and memory no longer depends on a grid. What you pay: fitting takes an optimisation per image, and the weights are opaque; no single weight tells you what is where.

1.5 Family three: a bit of both

Between the two lies a large family that keeps explicit parameters with a clear spatial meaning, and reads them out with something simple.

Hybrid representation. Explicit, spatially organised parameters, plus a cheap read-out rule or a tiny network that turns them into the signal at any location.
  • Feature grid plus a tiny network. Instant-NGP stores trainable vectors in multi-resolution tables and decodes them with a very small network[Müller et al., 2022].
  • A set of simple shapes. Store a list of basic primitives, each with a position, a size and a value, and a fixed rule for combining them at any location. No decoder network at all. The Gaussian is the most popular such primitive; it is the subject of the next chapter.

Your model belongs to this third family. Its main runs use 64 Gaussians per image. Each carries 384 feature numbers plus a handful for position and shape, so the whole set stores about a quarter of the numbers in the 16×16 grid, and it can still be read at any location.

Exactly how, and whether it deserves the word “implicit”, is the question Chapter 5 settles once you have the pieces.

1.6 Two questions that tell representations apart

When you meet a new method, ask:

  1. How is the signal read at a location? A lookup (explicit), a network evaluation (implicit), or a simple rule over stored parts (hybrid).
  2. How are the stored parameters found? Either by an optimisation run separately for each image, or by a network trained once on many images that outputs the parameters for a new image in a single pass.
Amortised. Parameters produced by a trained network in one pass, instead of by a separate optimisation per input. The cost of optimisation is paid once, during training, and “spread out” over all future inputs.
MethodRead-outParameters found by
Pixel gridLookupMeasurement
NeRF, SIRENNetwork at xOptimisation per signal
Instant-NGPGrid lookup + tiny networkOptimisation per signal
Your modelSimple rule over Gaussians (Chapters 2–4)Placement amortised by a network (Part III)

The second question is where your work differs most from the classic examples, all of which optimise each signal from scratch.

The two questions also give you your first defence line. If a reviewer says “this is just an implicit neural representation”, answer both: the signal is read by a simple rule over explicit parts that each sit somewhere in the image, and those parts are placed by one pass of a trained network, not by fitting each image. Size, speed and the kind of values stored do not decide the family.

1.7 Check yourself

Answer from memory; each question gives its reason once you choose.

1. A 32×32 grid of 384-number features is which kind of representation?

What matters is how the stored numbers are read. A grid is read by lookup, so it is explicit, whoever computed it.

2. You read a feature grid halfway between two cells with bilinear interpolation. What does the value you get tell you?

A grid holds only samples. Between them, the value is whatever your rule says, and the grid cannot tell you whether it is right. That is why a coarse grid can hide detail that no rule recovers.

3. What does an implicit representation give you that a grid does not?

A neural field can be evaluated anywhere. It does need per-signal optimisation (in the classic form), and its weights are not tied to locations.

4. What makes a method amortised?

Amortisation is about how parameters are found: one forward pass of a network trained on many inputs, instead of an optimisation per input.

5. Which pair of questions separates representations?

Size, speed, dimension and content vary within every family. Read-out and the way parameters are found define the family and the training regime.

6. A reviewer says your model is “just an implicit neural representation”. Which reply answers with the two questions?

The read-out is a simple rule over located, explicit parts, and a trained network places them in one pass. Storing fewer numbers does not change the family, and the read-out is not a network.

1.8 Close the loop

Without scrolling up, describe what a representation of an image is, using the words explicit, implicit and amortised. Then compare with your first answer.

A model answer

“A representation is a way of storing an image's signal. Explicit ones store samples and read them by lookup; implicit ones store a function that can be evaluated anywhere; hybrids store spatially meaningful parts and read them with a simple rule. Separately, the parameters are either optimised per image or produced in one pass by a trained network, which is called amortised. My model is a hybrid: Gaussians with a simple read-out, placed by an amortised network.”

Read next Xie et al., “Neural Fields in Visual Computing and Beyond” (2022), sections 1 and 2. A survey that defines the vocabulary this chapter used. Skim the later sections; you will meet their topics in order.

Something unclear, or want to rehearse how you would say this to a reviewer? Ask your teacher in the chat.

One-page summary of this chapter ·