What Does the Future Buy?
Future-predictive LiDAR pre-training bundles a learned transformation with supervision from future scans. This study separates the two over a shared scaffold. In the few-shot detection settings tested here, a present-only projector transfers competitively at lower cost; future targets make motion more accessible in a frozen readout without a stable downstream advantage.
The question
A LiDAR encoder can be pre-trained by asking it to predict what the sensor will see a moment from now. Doing so requires two separate additions: a learned transformation that carries the encoder’s features forward in time, and a loss computed against a scan recorded later. They are usually introduced together and reported together.
Only the encoder is kept. So the question this work asks is narrow and answerable:
Does the downstream detector benefit from the future supervision, or from the machinery that was added to deliver it?
The bounded answer, in the settings tested here: a present-only projector transfers competitively, while future supervision brings no stable detection gain we can resolve. Switching the future-target loss off with the recurrent architecture held fixed yields comparable detection at lower cost. Using the simpler, action-free projector cuts the 48-epoch pre-training cost by about 47% relative to Forecasting, but also changes the architecture. Future supervision is not inert: it measurably changes what the retained encoder exposes about object motion, without a matching improvement in the detector’s velocity estimates.
Everything below is bounded to two datasets, two detectors, few-shot fine-tuning regimes, and one pre-training run per arm. That last constraint is the main limit on how far any of it generalises, and it is stated again at the end.
This page is the synthesis. Four companion pages carry the parts that would interrupt it: the background, the implementation observations, the full protocol and result tables, and the probe constructions.
The test bed
The experiments are built on TREND, a future-predictive LiDAR pre-training method published as a NeurIPS 2025 Spotlight. It is used here as a test bed, not as a target: it is a clear, recent instance of the design being examined, with a public implementation to build a scaffold on.
The pre-training graph masks part of a sweep, encodes what is left, transforms the resulting latent state forward in time under the ego vehicle’s recorded motion, and renders a predicted distance along query rays. The loss compares those distances against the present frame and, under TREND’s objective, against a future frame too.
Pre-training
- InputLiDAR sweepPoints in the ego frame. Ten aggregated sweeps on nuScenes; a single 0.1-second scan on ONCE.
- InputMaskingPart of the input is hidden from the encoder. Query rays sampled from the full observation provide reconstruction targets.
- KeptEncoderSparse 3D backbone. The only component that reaches the downstream detector.
- DiscardedLatent transformationA recurrent step, conditioned on ego motion, that rolls the latent state forward in time.
- DiscardedNeural field rendererDecodes latent features into a predicted distance along each query ray.
- SupervisionRange lossesRays from the present frame, and under TREND's objective also rays from a future frame.
A single query ray leaves the sensor and meets a surface. The renderer predicts a distance along that ray; the loss is the gap between the predicted and the measured distance.
What a range target is. Each query ray leaves the sensor in a known direction. The renderer returns one number for it, the distance to the first surface it expects to meet, and the loss compares that number with the distance the sweep actually recorded. A future range target is the same quantity measured from a later position of the same vehicle.
New to self-supervised pre-training?
The detector is eventually trained on labelled boxes, which are expensive. Pre-training spends unlabelled sweeps first, on a task whose answer is already in the data: here, reproducing distances that the sensor measured. The hope is that the encoder learns something structural along the way, so that far fewer labels are needed afterwards. The parts added purely to make that task solvable are thrown away before the labelled stage begins, which is what makes “which part helped?” a question that needs an experiment.
A note on novelty: a projection head sitting between the representation and the loss is a standard device in self-supervised learning, not something introduced here or in TREND. What is at stake is not who proposed the head, but whether the reported benefit of predicting the future is attributable to the future targets or to that interface.
Read more Self-supervised pre-training and the TREND test bed works through the two-stage recipe, the pretext task step by step, and the projection-head literature this control is built against.
What the released path actually optimises
Before running anything, the scaffold had to be pinned down, and one detail did not match expectation. The paper describes the present frame as being reconstructed from the encoder’s own features, with the latent transformation standing between the encoder and the future frame. In the released implementation we inspected, both losses read the transformed features.
As the paper describes it
The present frame is rendered from the encoder's own features. The latent transformation stands between the encoder and the future frame only.
The encoder feeds the present-frame range loss directly, and also feeds the latent transformation, which feeds the future-frame range loss.
As the released implementation routes it
In the code we inspected, both losses read the transformed features. The encoder reaches neither loss without passing through the discarded head.
The encoder feeds the latent transformation, and the transformation feeds both the present-frame and the future-frame range loss.
Three things to keep apart. The paper's description, the behaviour of the released code, and the arms compared in this study are not the same object. Rather than guess which one the published numbers were produced with, the experiments fix a single scaffold and change one part of it at a time, including variants that restore the routing as described. Nothing here establishes what any unreleased training run did.
This matters for interpretation, not for attribution. If every loss reaches the encoder only through the head, then the head is always in the gradient path, and the encoder is never asked to be render-ready by itself. Which of these graphs any particular published number was produced with is not something this work can determine, and it does not try to.
Two things worth checking
Two further observations concern the future-frame target specifically. Both were found by running the released configuration and looking at what the future loss was actually being computed on.
nuScenes
Almost no future rays survive the filter
A bar representing the 6144 rays sampled for the future frame, of which roughly four pass the validity filter.
In the released configuration we ran, the validity mask applied to the future frame leaves on the order of four usable rays out of the 6144 sampled. The future term is present in the loss, and contributes almost no gradient. The sliver above is drawn wide enough to be visible; at true scale it would be a fraction of a pixel.
ONCE
The future frame lands in the wrong place
Schematic of a future scan overlaid on the present one: rotated and offset as inspected, closely overlapping once the pose handling is corrected.
On ONCE the two frames are related through recorded pose. In the path we inspected the future scan arrives rotated and displaced relative to the present one, so the future target asks the model to reproduce geometry that is not where it is claimed to be. Correcting the pose handling brings the two frames into visibly better agreement. The panels above are a schematic of that failure mode, not a rendered sample.
Why this is here. Neither observation is a result, and neither is offered as an explanation of anyone's published numbers. They are the reason this study stops reading the future-prediction objective off an existing implementation and instead builds a scaffold where the future term can be switched on and off deliberately.
These are the motivation for the rest of the page. An ablation that removes future supervision is only informative if future supervision was doing something in the first place, so the comparison is rebuilt from a scaffold where that can be checked rather than assumed.
Read more What the future objective actually sees gives the routing difference, the target geometry on both datasets, how each was checked, and what none of it can establish about unreleased runs.
One scaffold, one change at a time
Four pre-training arms share a scaffold, a data split and a budget. Moving between adjacent arms changes the ingredients shown below; the middle step bundles two changes.
- DirectNo added interfaceThe renderer reads the encoder output. Present-frame rays only.
- adds an interfaceone change
- ProjectionFeed-forward projectorAn action-free head sits between encoder and renderer. Present-frame rays only.
- adds recurrence and ego-motion conditioningtwo changes, bundled
- Recurrent ProjectionRecurrent, ego-motion conditionedTREND's head, with the future term switched off. Present-frame rays only.
- adds the future range lossone change
- ForecastingTREND's objectiveThe same recurrent head, supervised on present and future rays.
Only the encoder transfersEvery arm shares one scaffold, one dataset split and one pre-training budget. Everything the edges add lives to the right of the encoder and is deleted before fine-tuning, so each comparison asks what a discarded part left behind.
Reference. A randomly initialised detector, fine-tuned on the same data, marks where the whole ladder starts.
Side checks. Three further variants are run but not treated as coequal arms: a projector at half internal width, a head trained without the present-frame anchor, and a paper-like variant that reads the present loss from the encoder rather than through the head.
Read more Controlled comparisons and detector results sets out the shared protocols, the matched checkpoint trajectories in full, the compute accounting and the ONCE seed tables.
What the detector gets
The cleanest comparison holds the recurrent architecture fixed and switches only the future-target loss. Both pre-training trajectories are plotted in full, so each arm’s own best checkpoint is visible rather than hidden behind a single epoch chosen for the comparison.
Read at each arm’s own best checkpoint, Forecasting at pre-training epoch 24 and Projection at epoch 36, the two arms separate by 0.20 mAP, inside the spread of the fine-tuning executions themselves. Projection has higher NDS and lower mAVE in all three of its executions.
- +0.20mAP for Projection32.94 against 32.74
- +0.56NDS for Projection46.70 against 46.14
- ≈47%less compute per epoch98 against 186 GPU-hours over 48 epochs
ONCE supplies a second dataset and a second detector, with three distinct fine-tuning seeds per cell. The selected nuScenes checkpoints above also have three independent fine-tuning executions, though those are not three distinct seed labels throughout.
The 59.26 AP figure originally recorded for Forecasting at epoch 12 turns out to be the high realization of its cell; the three-seed mean is 58.30 ± 0.84. At epoch 24 the three arm means are 58.24, 58.25 and 58.21.
These spreads overlap, and overlapping spreads are a failure to separate rather than a demonstration that the arms are equal. With three seeds, only fairly large differences would have been detectable. What ONCE supports is the negative form of the claim: across two datasets and two detectors, no stable advantage for future supervision was found. The single scratch run supports the schedule choice, not a precise absolute pre-training gain.
The transformation edge points the other way. On nuScenes both arms peak at the same checkpoint, and Projection exceeds Direct by 1.25 mAP there and by 1.50 at epoch 48, with the same sign in NDS. One pre-training trajectory per arm limits the precision, but the direction is consistent with the discarded interface helping transfer independently of future targets.
What the discarded head does
If the interface matters, the next question is what it produces. A frozen-checkpoint probe reads mean voxel height linearly, once from the encoder output and once from the transformed features.
Mean height is only weakly accessible at the encoder output, around 0.11 R², and close to explicit after the head, around 0.94. The reading is that the head formats abstract encoder features into a geometric, render-ready code, and that recurrence and action conditioning are not what makes it do so.
This describes the interface. It does not establish that formatting is why the encoder behind it transfers better. That step would need an intervention on the head, not a readout of it.
What the retained encoder holds
The second probe looks at the part that survives. It pools frozen encoder features inside ground-truth car boxes on nuScenes and fits a linear speed readout on held-out scenes.
Speed is already partly accessible without any future supervision: Recurrent Projection reaches 0.367 R² and the action-free Projection 0.376. Forecasting raises it to 0.424, and the paired difference is positive on all five shared splits. Future supervision therefore does reach the retained encoder; it is not confined to the machinery that gets discarded.
That accessibility grows with the amount of future-supervised pre-training. The detector’s velocity error does not follow it down.
The result is a dissociation. Future targets reliably improve this frozen linear readout within the measured trajectory, and the detector evaluated here does not convert that into better velocity estimates. At epoch 48 the largest readout difference coincides with the largest observed car-AVE cost. Whether that endpoint generalises would need independent pre-training repeats, which do not exist here.
The same probe on ONCE
ONCE is a different regime, and the probe behaves differently there. Its encoder sees a single 0.1-second sweep with no time channel, so nothing in the input directly encodes displacement.
On ONCE, moving versus stationary cars are separated well above chance by every pre-trained arm and by the geometry control, while continuous speed is not reliably linearly decoded by any of them.
Show the numbers
| Arm (ONCE, PT epoch 24) | Moving vs stationary AUC | Speed R² |
|---|---|---|
| Box geometry only | 0.597 | 0.021 |
| Recurrent Projection | 0.680 | −0.049 |
| Forecasting | 0.696 | 0.021 |
| Projection | 0.699 | −0.176 |
Two things have to stay separate here.
Continuous speed is not reliably linearly decodable from these frozen features in this probe: the R² values are near zero or negative across the arms, and the same holds at pre-training epochs 12, 24 and 36. That is a statement about this readout: one linear model, one pooling scheme, one frozen checkpoint. It is not a claim that the features contain no motion information, and it is not an information-theoretic limit. A stronger or differently posed decoder could do better.
Moving versus stationary behaves differently: every arm sits well above chance, and so does the control that sees only box geometry. A parked car and a moving one differ in where they are, how large a return they give and what surrounds them, so a propensity score can be built from static context alone. The observed gap between the arms and the 0.597 control is modest, and it does not license reading “the encoder knows about motion” off the AUC column.
Read more Representation, motion probes and limits describes how each probe is built before what it scored, including the checkpoint trajectories this section reads one point from.
What this establishes, and what it does not
- One pre-training run per arm. Every error bar on this page measures fine-tuning or probe-split variation. None of them measures pre-training run-to-run variation, which is the source of variation most likely to matter.
- Selected checkpoints. The headline nuScenes pair was chosen after looking at detection results. The matched trajectories are the control for that, and they are why the trajectory figure comes first.
- Probe splits are not model seeds. Five paired nuScenes splits and block-level ONCE splits exclude split noise. They say nothing about how much the probe result would move under a different pre-training run.
- Detection only. Both datasets are evaluated through object detection. A task that consumes motion more directly was not tested, and the dissociation result is precisely the reason to expect that it might behave differently.
- Compute accounting. Forecasting’s cost is measured; the present-only costs are calibrated estimates for the intended objectives. The supplement records the accounting.
What survives all of that is a trade-off rather than a ranking: in these settings, a present-only projector transfers competitively at substantially lower pre-training cost. The higher NDS, the lower mAVE and the readout dissociation are observations within the evaluated setting, not estimates of a universal advantage.
The open question
Future supervision makes motion more accessible to a frozen linear readout, and the detector does not spend that information here.
Which driving tasks actually cash in future supervision, beyond what present-only pre-training already provides?
Motion segmentation and occupancy forecasting are the natural next places to look, because both consume motion in their output rather than as a side product of box regression.
A second direction is about what the target is made of. Everything here reconstructs observations: distances the sensor measured. Predicting latent features of a future frame instead, rather than its raw geometry, is a different design with a different failure mode, and it would sidestep the target-construction problems shown earlier, since there would be no future range image to filter or align. That comparison is not performed here; it is where this line of work points.
The printed version
The argument was compressed onto one A0 sheet for the ELLIS Summer School on Autonomous Driving in September 2026. It is a snapshot of that moment: the page above has since been extended, and where the two disagree, the page is the later account.
