DriveRT

Perception models for driving are sparse, dynamic and heterogeneous. Points are grouped into voxels, voxels go through sparse convolutions, features are projected to a bird’s-eye view, and the predictions are decoded and suppressed. DriveRT compiles this whole program into one .drt file and executes it without Python, PyTorch or the source framework.

• 1.85×faster than OpenPCDet with spconvVoxelNeXt, nuScenes, FP32
• Sameboxes, count and order as the source detectoron every qualification frame
• 851tests across compiler, Plan format and runtime
• Apache-2.0open source license

The problem

Compilers and inference engines are designed for dense tensors whose shapes are known in advance. Driving models do not match this. The number of voxels changes every frame, neighbours must be found before the convolution can run, and a large part of the model is not a neural network: voxelization, voxel reordering, bird’s-eye-view pooling, box decoding and rotated suppression.

As a result, each model is deployed by hand:

• NVIDIA’s LiDAR reference deployments split each model into separate parts: hand-written CUDA voxelization, a sparse convolution library for the backbone, TensorRT for the dense head, hand-written decoding and NMS, and per-model configuration files that repeat the training setup.
• Camera BEV models need their own custom TensorRT plugins (grid sampling, multi-scale deformable attention, BEV pooling), which are written, maintained and quantized outside the engine.
• Mixed precision needs additional manual work: TensorRT’s accuracy guidance asks users to export normalization layers with specific ONNX opsets or keep them in FP32 to avoid FP16 overflow.

This work is specific to each model and has to be repeated for the next one.

DriveRT is a single compiler for these models. Support is defined by the computations a model performs, such as a sparse convolution, a projection or a suppression, and does not depend on the model. A model can be deployed when all its computations are supported, and a computation supported for one model is available to every other model that uses it.

Whole-program optimization

When a deployment is split into separate engines, plugins and kernels, each part is optimized on its own and no optimization can span two parts.

DriveRT represents the sparse structure explicitly. Coordinates, active row counts, neighbour maps and sparse-to-dense transitions are values in one graph, together with the dense layers, the preprocessing and the postprocessing. This allows optimizations across the whole program.

Row-selective evaluation

Detection heads follow two designs. Query-based heads such as TransFusion rank a class heatmap first, keep a few hundred proposals, and run their regression layers only on those proposals. Center-based heads such as CenterPoint and VoxelNeXt compute every regression branch at every location, and the decoder reads the values only at the top-k peaks. The second design is simpler to write, and at inference it computes values that are not read.

DriveRT finds this structure in the graph. A producer is evaluated only at the selected rows when the compiler proves two things: the producer does not affect which rows are selected, and every use of it reads only the selected rows through pure, row-wise Operations. The model code does not change. The compiled Plan evaluates a center-based head the way a query-based head is written by hand, and if the proof does not hold, all branches run on all rows.

In VoxelNeXt, six of the 36 branches produce the class scores used for ranking. The other thirty run only at the kept rows, a few hundred out of up to about 15,000. Over five qualification frames this saves 7.6 ms, and with the same region input the outputs are bit-identical to evaluating every row.

The current kernels cover pointwise branches. CenterPoint has the same dataflow, but its regression branches are 3×3 convolutions that read neighbouring cells; running them only at the selected positions needs a convolution kernel that evaluates a list of positions, which is not implemented yet. The detection head notes describe the proof, the refusals and the measurements.

Shared neighbour maps

Building a neighbour map can cost more than the convolution that reads it. A map is identified by its coordinate domain, kernel geometry and row order, so all Steps that read the same domain use the same map.

Counts kept on the GPU

A strided sparse convolution produces a number of voxels that depends on the input. Counted execution keeps these counts on the device through the whole chain, which reduces four host synchronizations to one.

Kernels selected per GPU

Sparse convolution is implemented by a family of parameterized Algorithms. The 27 neighbours of a voxel form a bit mask; sorting rows by this mask lets GPU threads execute the same work, and grouping kernel offsets reduces the work of each launch. Tile sizes, bit orders and offset groups are configuration values, measured per layer on the target GPU. The sparse convolution notes describe each FP32 optimization and the designs that were rejected.

Compilation and deployment

The model is compiled from its PyTorch code without changes.

plan = drivert.compile(model, representative_inputs, profiles=profiles)
plan.save("model.drt")

runtime = load("model.drt", device);   /* C, C++ or Python */
outputs = run(runtime, inputs);

Deployment needs one file and one library. Python, PyTorch, OpenPCDet and spconv are not required at runtime, and no per-model CUDA code has to be written.

• Capture. DriveRT records the operations the model executes at the PyTorch dispatcher as straight-line SSA, and converts in-place writes into pure dataflow. Native calls that the dispatcher does not see, such as a spconv convolution or an NMS extension, are recorded at the function that performs them. Control flow that depends on input data is rejected with its source line.
• Canonical graph. Dense, sparse, preprocessing and postprocessing operations use one representation. Every dynamic dimension has a declared range.
• Operation, Algorithm, Step, Plan. An Operation describes what is computed. An Algorithm is one backend implementation, stored as data: name, precision and configuration. A Step is one or more Operations with one selected Algorithm. A Plan contains the graph, the Steps and the weights.
• Verified grouping. Operations are grouped into one Step only when the graph shows that no intermediate value is used outside the group. Otherwise the Operations stay separate, or compilation fails with the Operation name.
• Complete coverage. An unsupported Operation is a compile error that names it. A Plan does not fall back to the source framework.
• Runtime. Loading resolves values, maps and weights into integer slots and prepared layouts once. Execution calls a fixed array of function pointers. The runtime exposes a C ABI, with thin C++ and Python wrappers.
• Dynamic sizes. Buffers start from an estimated capacity. If a buffer is too small, the invocation is repeated with a larger buffer. Truncated results are not returned.

Measurement and reproducibility

• Every Algorithm is tested against a reference implementation before it is timed.
• Algorithms are selected offline on the target GPU, using the same execution path as deployment, and the selection is stored in a documented file. The same inputs produce a byte-identical Plan.
• Performance is measured for single Algorithms, single Steps and the whole Plan, reporting P50, P95 and P99, kernel launches, host synchronizations and memory.
• Rejected designs are kept in the project record with their measurements, for example compact pair kernels, fused two-convolution layouts, block-dense hybrids and exact mask-run scheduling.

Inspection and debugging

TensorRT’s builder, runtime and kernels are closed source. When an FP16 model loses accuracy or a build changes between versions, the cause can only be searched from outside the engine, and it cannot be fixed by the user.

DriveRT is open at every level:

• Artifact contents. drivert inspect model.drt lists inputs, profiles, Operations, Steps and the selected Algorithms, without a GPU or Python.
• Errors. Errors report the Operation, Step, Algorithm, input or rule involved.
• Correctness. Reference tests for every Algorithm, retained intermediate values and per-Step timing.
• Extension. A new Algorithm is one catalog entry, one binder, one kernel, one test and one measurement.
• Documentation. Every Operation family has its own document.

Precision is not changed implicitly, a selected kernel is not replaced at load time, and outputs are not truncated.

Status

Results

VoxelNeXt on nuScenes, RTX 4060 Laptop GPU, FP32. The complete detector runs from raw points to boxes through the standalone C library, with the same boxes, count and order as the source detector on every qualification frame.

In progress

• Mixed TF32 execution: qualified on the reference model, about 15% faster, not yet enabled by default.
• Merging the remaining CUDA kernels into one framework-independent source.
• Freezing the Plan format and the C ABI for a first release.

Next

• FP16 and INT8, each with its own correctness tests.
• Offline retargeting of a Plan to another GPU.
• A ROS adapter over the same C ABI.
• Open-source release.

Contributing

DriveRT is released under Apache-2.0. A computation added for one model can be used by every model that contains it. Researchers and engineers working on perception models, deployment or GPU kernels are welcome to get in touch. The source code lives at github.com/LucaBrembilla/DriveRT.

• Source code (GitHub)
• Contact