Representation, motion probes and limits

Why probe at all

Detection accuracy is a coarse readout. It answers whether an initialisation helps, not what changed inside the network, and on this ladder several arms land in the same band. The probes below look at two specific places: the head that pre-training discards, and the encoder that fine-tuning keeps.

Every probe here runs on a frozen checkpoint. Nothing is trained except a linear readout, and the readout is always fitted and evaluated on disjoint data. A probe result is a statement about what is linearly accessible in a representation, which is narrower than a statement about what the representation contains.

The discarded head: a geometric recoding

Construction

Take a frozen pre-trained checkpoint and run it forward on validation data. Two feature volumes come out of the same forward pass: the encoder output, and the same volume after the latent transformation has been applied.

For each occupied feature voxel, compute a target from the raw LiDAR points assigned to it. The main target is the mean height of those points. Two further targets are the logarithm of the point count and the mean intensity.

Fit a ridge regression from features to target on 70% of the voxels and report R² on the held-out 30%. Run the identical fit twice, once from the encoder output and once from the transformed volume. The comparison between the two numbers is the whole measurement.

Result

Mean height is only weakly accessible to this linear readout at the encoder output, around 0.11, and much more accessible after the head, around 0.94. This establishes a change in the accessibility of the measured geometric target, not preservation of every other property.

Three things follow, and one does not.

The head behaves as a formatter. It converts the encoder’s abstract representation into an explicitly geometric, render-ready code, which is what a renderer needs and what a detector does not.

The effect does not require recurrence or ego-motion conditioning in the tested arms. One action-free spatial projection reaches a similar readout to the recurrent, action-conditioned head, and halving the projector’s internal width does not reduce this particular score.

The arm without a present-frame anchor has the lowest after-head score in this table and weak transfer. This is consistent with the anchor supporting the geometric readout, though the probe alone does not isolate a causal mechanism.

What does not follow is causation. This describes the interface; it does not establish that formatting is why the encoder behind it transfers better. That step needs an intervention on the head, not a readout of it.

Intensity stays undecodable in every arm, which is the control working as intended: the intensity term carries zero weight in the loss, so nothing ever supervised it.

The action-sensitivity control

Construction

Keep one encoder output fixed. Run the frozen transformation on it three times: once with the true recorded ego action, once with the action replaced by zeros, and once with another sample’s valid action.

Score the change in the entire output tensor caused by the intervention, divided by the transformation’s total change away from its input, and average over 20 batches. Zero means the action is ignored. One means the action explains a change as large as the whole transformation.

Result

Future supervision is what makes the head motion-conditioned, and only when the present-frame anchor is also there. Without the future loss the recurrent head barely consults the action at all, which is worth holding next to the detection results: the arm that ignores the action reaches the same transfer band as the arm that uses it. Replacing the action with another sample’s is a weak intervention here, since actions within a batch are similar, so the zeroed column is the one to read.

The retained encoder on nuScenes: speed

Construction

This probe measures the module fine-tuning keeps, not the one it discards.

Extract the frozen encoder’s fourth-stage sparse feature map on unmasked input, with augmentation switched off so the network input and the annotated boxes share one frame. Pool features inside ground-truth car boxes. 300 keyframes yield 3,084 car objects across 147 scenes.

Fit two linear readouts: moving against stationary at a 0.2 m/s threshold, with 32.9% of objects moving, and continuous speed. Splits are at scene level, so adjacent frames and repeated tracks cannot leak across them, and every arm is scored on exactly the same objects and the same five splits. Per-split differences between arms are therefore paired, and the paired sign count is the number to read rather than each arm’s own spread.

The control is a readout from box centre, range, size, yaw and voxel count with no encoder features at all. An arm has to beat it before anything representational can be claimed.

Result at a matched checkpoint

The probe is sensitive: every arm sits far above the geometry control, so the aggregated multi-sweep input carries motion and the encoder retains it. Speed is already partly accessible without any future supervision. Forecasting adds to it, and the paired difference against Recurrent Projection is positive on all five splits.

Dose, and where it stops agreeing with the detector

The same probe run at four matched pre-training checkpoints, against the downstream car velocity error from the fine-tuning cells of those same checkpoints.

The encoder side is clean: monotone over four points, all five paired splits at every epoch, and at the two mature checkpoints the split spread of about 0.013 is a fraction of the gap. Absolute levels rise for every arm, so pre-training itself adds motion information and the future loss adds an increment on top of it.

The downstream side is not monotone, and its middle points sit inside the measured fine-tuning noise floor of 0.010 to 0.014 on car AVE. The defensible statement is the dissociation plus the encoder-side dose curve, not two opposed monotone curves. At epoch 48 the largest readout advantage coincides with the largest velocity cost, and whether that endpoint generalises would need independent pre-training repeats, which do not exist here.

The retained encoder on ONCE: a negative control that works

Construction

ONCE ships no velocity and no track identifiers, so motion labels have to be derived. Annotated sequences carry boxes at 1 Hz and every frame carries a pose, which is enough.

Box centres are lifted into the scene frame under the corrected pose convention and matched to same-class boxes across the one-second annotation step, behind a distance gate, a size gate, a second-nearest ratio test and a forward-and-backward mutuality requirement. Anything ambiguous is dropped, because a wrong match manufactures motion out of an identity switch: 45,651 objects are kept out of 56,695 raw matches.

The labels are checked before they are used. Per-class median speeds come out physically right: pedestrian 0.74 m/s, cyclist 2.53, bus 2.47, truck 3.08, car 0.17, with 51.9% of objects below 0.3 m/s forming the parked cluster. Self-consistency across overlapping frame pairs has a median of 0.14 m/s, so annotation jitter sits well clear of the moving threshold.

Only four labelled sequences exist, so splits are 60 contiguous one-minute blocks rather than scenes. They are less independent than the nuScenes scene splits. The probe scores 3,204 cars.

The encoder input matters more here than anywhere else on this page. On ONCE it is a single 0.1-second scan with no aggregation and no time channel. Extra frames enter pre-training only as rendering targets, never as encoder input. Unlike the aggregated nuScenes input, it provides no direct cross-frame displacement measurement.

Result

No ONCE arm yields a useful continuous-speed readout with this probe. Speed R² is near zero or negative, where negative means the readout does worse than predicting the mean, against 0.37 to 0.42 on nuScenes at the same epoch with the same metric. That is an empirical contrast between these inputs and protocols, not an information-theoretic ceiling.

The binary task behaves differently. At epoch 24 every arm sits 0.091 to 0.103 AUC above the geometry control, on all five splits, and every arm stays above the control at every dose. That signal is real, but it does not by itself demonstrate velocity estimation. A parked car and a moving one differ in where they are, how large a return they give and what surrounds them, so a propensity score can be built from static context alone. The encoder knows more than box centre, size and yaw; static context is one plausible source of the additional AUC.

The dose curve, and the near-tie at epoch 24

Moving-against-stationary AUC. Read on its own, epoch 24 says future supervision adds nothing to the binary task. Read across the dose range, epoch 24 is the outlier: the two neighbouring checkpoints both give a consistent advantage on all five splits. The statement that holds is the one over three doses, and epoch 24 is reported here rather than dropped because a single cell is what would have produced the wrong one.

Speed R² stays near zero or negative across the whole ONCE dose range for every arm. Over its own dose range nuScenes runs 0.278 to 0.426 for Recurrent Projection and 0.325 to 0.536 for Forecasting. On a single-sweep input the future loss may be sharpening static-context cues instead of making speed linearly accessible in this readout. Predicting the next frame under ego motion could reward cues correlated with the parked-against-in-lane distinction a static snapshot already supports. That mechanism remains an interpretation, not something this probe isolates.

ONCE cannot test the downstream velocity dissociation either way: its detection metric contains no velocity term, and this probe does not recover useful continuous speed from its single-sweep input. The flat ONCE detection results are consistent with that boundary, not explained by it alone.

What this establishes, and what stays open

Established, inside the settings tested:

• The discarded head performs a geometric recoding, and that recoding does not require recurrence, ego-action conditioning, or full internal width.
• Future supervision leaves more linearly accessible motion information in the retained encoder, with the architecture held fixed, monotonically in the amount of it, on all five paired splits at every checkpoint.
• The detector evaluated here does not convert that into better velocity estimates.
• On the single-sweep ONCE input, this linear probe finds a binary moving-versus-stationary signal but no useful continuous-speed readout. Static context is a plausible contributor, not a proven exclusive explanation.

Limits that apply to all of it:

• One pre-training run per arm. Every interval on this page is split variation or fine-tuning variation. None of it prices pre-training run-to-run variation.
• Linear accessibility only. A negative or near-zero R² means this readout fails, with one linear model, one pooling scheme and one frozen checkpoint. It is not an information-theoretic limit, and a stronger or differently posed decoder could do better. Not linearly decodable by this probe is a different statement from no motion information.
• Scope. The nuScenes probe covers cars only. The ONCE splits are one-minute blocks from four sequences, not independent scenes.
• Detection only downstream. Both datasets are evaluated through object detection.

Open, and testable:

• A task that consumes motion in its output. Motion segmentation and occupancy forecasting both do, and the dissociation is exactly the reason to expect them to behave differently from box regression. Nothing here says they will.
• Latent future-feature prediction. Predicting features of a future frame instead of its raw geometry is a different design, and it sidesteps the target construction problems described earlier, since there would be no future range image to filter or align. It has not been tested here and no result on this site bears on it.

Back The study overview carries the summary of all four pages.