3D Gaussian Splatting for Real-Time Radiance Field Rendering
Click a node to select
Main topic Killer Application
Real-time ($\geq$ 30 fps), 1080p novel-view synthesis at state-of-the-art radiance-field quality — the first method to deliver both at the same time.

Until this paper the field had a hard trade-off. The quality leader, Mip-NeRF360, needed 48 h of training and rendered at 0.06 fps. The speed leaders, InstantNGP and Plenoxels, trained in minutes but rendered at 9–17 fps at visibly lower quality. This work reports 0.815 SSIM / 27.21 PSNR at 134 fps on the Mip-NeRF360 dataset after 41 min of training — matching or beating the 48-hour method while rendering roughly 2000$\times$ faster.

Three contributions get it there, and each is remarkable on its own:

1. Anisotropic 3D Gaussians as the scene representation. A primitive that is simultaneously a differentiable *volumetric* element (so it optimizes like a radiance field) and an *explicit, unstructured* one (so it rasterizes like a point cloud). No voxel grid, no hash table, no space compaction — and no wasted compute in empty space.

2. Optimization interleaved with adaptive density control. The set of Gaussians is not fixed: it is cloned, split and pruned during training based on the view-space positional gradient, so the representation grows exactly where the reconstruction is wrong. 1–5 million Gaussians covers every scene tested.

3. A visibility-aware differentiable tile rasterizer. Sorting is done once per frame with a single GPU radix sort over (tile, depth) keys instead of once per pixel, and the backward pass places no limit on how many splats receive gradients. The same kernel makes training fast *and* rendering real-time.

A fourth thing worth naming: there is no neural network anywhere in the pipeline. Geometry is explicit Gaussians, view-dependent color is spherical-harmonic coefficients, rendering is rasterization. The paper states outright that this contradicts the accepted wisdom that a continuous (implicit) representation is needed for quality.

fig01_teaser.png

Fig. 1 is the claim in one strip: same scene, same held-out view. InstantNGP reaches 22.1 PSNR at 9.2 fps, Mip-NeRF360 reaches 24.3 PSNR after 48 h at 0.071 fps, and this method reaches 23.6 PSNR in 6 minutes at 135 fps, or 25.2 PSNR in 51 minutes at 93 fps. The left half of the figure is the old trade-off; the right half is the point of the paper.
Original Abstract, p.1
We introduce three key elements that allow us to achieve state-of-the-art visual quality while maintaining competitive training times and importantly allow high-quality real-time ($\geq$ 30 fps) novel-view synthesis at 1080p resolution.
Main topic Why It's Needed
Every prior family of methods fails on one axis of the (quality, training time, render speed) triangle, and the failure is structural rather than an implementation detail.

NeRF-style implicit fields. Color along a ray is accumulated by volumetric ray-marching:

$$C = \sum_{i=1}^{N} T_i\,\bigl(1 - \exp(-\sigma_i \delta_i)\bigr)\,\mathbf{c}_i \quad\text{with}\quad T_i = \exp\!\left(-\sum_{j=1}^{i-1} \sigma_j \delta_j\right)$$

Because the field is continuous and implicitly encodes where matter is, the samples $i$ have to be found by stochastic sampling — expensive, and a source of noise. Quality-optimal Mip-NeRF360 pays 48 h of training and 10 s per frame for this.

Fast grid-based fields (InstantNGP, Plenoxels). These shrink or delete the MLP by storing features in voxel or hash grids. But they still ray-march, so many samples per ray are still required, and the accelerating grid is itself a structured prior: the paper notes they struggle to represent empty space and that image quality is limited by that grid.

Point-based rendering. Explicit and fast, but prior neural point methods depend on Multi-View Stereo geometry and therefore inherit its over- and under-reconstruction artifacts on featureless, shiny or thin regions, and their CNN rendering stage causes temporal flickering. Pulsar rasterizes fast but is order-independent and caps how many splats receive gradients.

The gap this creates. Nobody had an explicit representation that could be *optimized* as well as a volumetric one. That is the hole the paper walks into.
Original §1, p.1
While the continuous nature of these methods helps optimization, the stochastic sampling required for rendering is costly and can result in noise.
Main topic Solution
Input is exactly what a NeRF pipeline already needs: photos of a static scene plus SfM-calibrated cameras. The sparse point cloud SfM produces as a by-product — normally discarded — becomes the initialization.

fig02_pipeline.png

Fig. 2 shows why the three contributions are one system rather than three tricks. The differentiable tile rasterizer sits on the *gradient* path (blue), not just the display path: it is what makes the projection, the Gaussian parameters and the density-control decisions all trainable from an image loss. Take it out and neither the optimization nor the real-time rendering survives.

Component 1 — the primitive. A 3D Gaussian with mean $\mu$ and full covariance $\Sigma$:

$$G(x) = e^{-\frac{1}{2}(x-\mu)^{T}\Sigma^{-1}(x-\mu)}$$

It projects to a 2D splat in camera space by $\Sigma' = J W \Sigma W^{T} J^{T}$, and $\Sigma$ is stored not directly but as a scale vector $s$ and a rotation quaternion $q$, so gradient descent can never produce an invalid (non positive semi-definite) covariance:

$$\Sigma = R S S^{T} R^{T}$$

Component 2 — optimization with adaptive density control. SGD on position, opacity $\alpha$, covariance and SH coefficients against

$$\mathcal{L} = (1-\lambda)\mathcal{L}_1 + \lambda \mathcal{L}_{\text{D-SSIM}}, \qquad \lambda = 0.2$$

every 100 iterations Gaussians whose view-space positional gradient exceeds $\tau_{\text{pos}} = 0.0002$ are cloned (if small: under-reconstruction) or split (if large: over-reconstruction), and near-transparent or oversized ones are pruned.

Component 3 — the tile rasterizer. Screen is cut into $16 \times 16$ tiles; Gaussians are frustum-culled at 99% confidence, duplicated once per overlapped tile, given a 64-bit key packing tile ID in the high bits and depth in the low bits, and sorted once by a single GPU radix sort. Each tile is then one thread block blending front-to-back out of shared memory until $\alpha$ saturates. The backward pass re-walks the same sorted lists back-to-front, so gradients reach an unbounded number of splats at constant per-pixel memory cost.
Original §3, p.4
The input to our method is a set of images of a static scene, together with the corresponding cameras calibrated by SfM [Schönberger and Frahm 2016] which produces a sparse point cloud as a side-effect.
Main topic Results
13 real scenes (all 9 Mip-NeRF360 scenes, 2 from Tanks&Temples, 2 from Deep Blending) plus the synthetic Blender set, one hyperparameter configuration for everything, all timings on a single A6000.

Table 1 — Mip-NeRF360 dataset ($\dagger$ = numbers copied from the original publication):
MethodSSIM $\uparrow$PSNR $\uparrow$LPIPS $\downarrow$TrainFPSMem
Plenoxels0.62623.080.46325m49s6.792.1GB
INGP-Base0.67125.300.3715m37s11.713MB
INGP-Big0.69925.590.3317m30s9.4348MB
Mip-NeRF3600.792$\dagger$27.69$\dagger$0.237$\dagger$48h0.068.6MB
Ours-7K0.77025.600.2796m25s160523MB
Ours-30K0.81527.210.21441m33s134734MB

Read the table against the Killer Application rather than column by column. Ours-7K vs INGP-Big is the speed claim: comparable training time, clearly better SSIM/LPIPS, and 17$\times$ the frame rate. Ours-30K vs Mip-NeRF360 is the quality claim: better SSIM and LPIPS, 0.5 dB behind on PSNR, at 1/70th the training time and 2000$\times$ the frame rate. No prior row in the table is simultaneously above 0.79 SSIM and above 30 fps; that empty quadrant is exactly what the paper set out to fill.

On Tanks&Temples and Deep Blending the margin is larger — Ours-30K wins SSIM, PSNR *and* LPIPS outright on both (see toggles), which matters because those captures are the messy, unbounded, real-world kind that grid methods handle worst.
Original §7.2, p.9
The table shows that our fully converged model achieves quality that is on par and sometimes slightly better than the SOTA Mip-NeRF360 method; note that on the same hardware, their average training time was 48 hours, compared to our 35-45min, and their rendering time is 10s/frame.
Table 1 (cont.) — Tanks&Temples and Deep Blending
Tanks&Temples

| Method | SSIM $\uparrow$ | PSNR $\uparrow$ | LPIPS $\downarrow$ | Train | FPS | Mem |
|---|---|---|---|---|---|---|
| Plenoxels | 0.719 | 21.08 | 0.379 | 25m5s | 13.0 | 2.3GB |
| INGP-Base | 0.723 | 21.72 | 0.330 | 5m26s | 17.1 | 13MB |
| INGP-Big | 0.745 | 21.92 | 0.305 | 6m59s | 14.4 | 48MB |
| Mip-NeRF360 | 0.759 | 22.22 | 0.257 | 48h | 0.14 | 8.6MB |
| Ours-7K | 0.767 | 21.20 | 0.280 | 6m55s | 197 | 270MB |
| Ours-30K | 0.841 | 23.14 | 0.183 | 26m54s | 154 | 411MB |

Deep Blending

| Method | SSIM $\uparrow$ | PSNR $\uparrow$ | LPIPS $\downarrow$ | Train | FPS | Mem |
|---|---|---|---|---|---|---|
| Plenoxels | 0.795 | 23.06 | 0.510 | 27m49s | 11.2 | 2.7GB |
| INGP-Base | 0.797 | 23.62 | 0.423 | 6m31s | 3.26 | 13MB |
| INGP-Big | 0.817 | 24.96 | 0.390 | 8m | 2.79 | 48MB |
| Mip-NeRF360 | 0.901 | 29.40 | 0.245 | 48h | 0.09 | 8.6MB |
| Ours-7K | 0.875 | 27.78 | 0.317 | 4m35s | 172 | 386MB |
| Ours-30K | 0.903 | 29.41 | 0.243 | 36m2s | 137 | 676MB |

Note the Deep Blending row for Ours-7K: 27.78 PSNR in 4m35s, already beating every non-Mip-NeRF360 baseline including ones that trained longer. The 30K column then matches the 48-hour method to within 0.01 dB.
Table 2 — Synthetic NeRF (PSNR), from 100K random points
| Method | Mic | Chair | Ship | Materials | Lego | Drums | Ficus | Hotdog | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| Plenoxels | 33.26 | 33.98 | 29.62 | 29.14 | 34.10 | 25.35 | 31.83 | 36.81 | 31.76 |
| INGP-Base | 36.22 | 35.00 | 31.10 | 29.78 | 36.39 | 26.02 | 33.51 | 37.40 | 33.18 |
| Mip-NeRF | 36.51 | 35.14 | 30.41 | 30.71 | 35.70 | 25.48 | 33.29 | 37.48 | 33.09 |
| Point-NeRF | 35.95 | 35.40 | 30.97 | 29.61 | 35.04 | 26.06 | 36.13 | 37.30 | 33.30 |
| Ours-30K | 35.36 | 35.83 | 30.80 | 30.00 | 35.78 | 26.15 | 34.87 | 37.72 | 33.32 |

The interesting part is not the 0.02 dB average win — it is the starting condition. This run begins from 100K uniformly random Gaussians, no SfM points at all, and adaptive density control prunes them to 6–10K meaningful ones before growing back to 200–500K. That is direct evidence that the density-control loop, not a good initialization, is what builds the representation. These scenes render at 180–300 FPS.
Table 3 — Ablations (PSNR)
| Variant | Truck-30K | Garden-30K | Bicycle-30K | Average-5K | Average-30K |
|---|---|---|---|---|---|
| Limited-BW | 13.84 | 22.88 | 20.87 | 19.16 | 19.19 |
| Random Init | 18.02 | 22.19 | 21.05 | 19.17 | 20.42 |
| No-Split | 20.59 | 26.11 | 25.02 | 21.50 | 23.90 |
| No-SH | 24.39 | 26.59 | 25.08 | 23.48 | 25.35 |
| No-Clone | 24.82 | 27.47 | 25.46 | 23.35 | 25.91 |
| Isotropic | 23.89 | 27.00 | 24.81 | 23.56 | 25.23 |
| Full | 24.81 | 27.70 | 25.65 | 23.90 | 26.05 |

Ranked by damage done, the load-bearing choices are: Limited-BW (capping gradients to the 10 front-most splats, $-6.9$ dB average, $-11$ dB on Truck), Random Init ($-5.6$ dB), No-Split ($-2.2$ dB), then anisotropy, SH and cloning at roughly $-0.1$ to $-0.8$ dB. Note that Random Init still produces a usable scene — the method degrades rather than collapses without SfM points, which is why the synthetic-scene experiment above works at all.
Main topic Conclusion
What it enables. Radiance fields stop being an offline format. A capture can be optimized in tens of minutes and then navigated interactively at 1080p in a real viewer (the authors built one on the open-source SIBR framework and measured their frame rates through it). Because the output is an unstructured set of explicit primitives in ordinary Euclidean space — no space warping, no compaction, no network weights — it drops into a conventional GPU rasterization pipeline rather than requiring a bespoke one.

The conceptual result. The paper's own framing is that it refutes a belief, not just a benchmark: a *continuous* representation is not strictly necessary for fast, high-quality radiance-field training. What volumetric rendering actually needed was differentiability and the ability to create/destroy/move matter — not continuity. 3D Gaussians supply both while remaining rasterizable.

Honest limitations the authors state. Artifacts in poorly observed regions; elongated or splotchy Gaussians; popping when large Gaussians switch blending order, caused by the guard-band culling and the approximate per-tile visibility ordering; no regularization at all; the position learning rate may need lowering for very large (urban) scenes; and peak training memory above 20 GB with several hundred MB per trained scene — far heavier than a NeRF's 8.6–48 MB.

Where the headroom is. About 80% of training time is still Python/PyTorch — only rasterization is CUDA. And the authors flag mesh reconstruction from the Gaussians as the interesting open question, since it would locate the method precisely on the continuum between volumetric and surface representations.
Original §8, p.11
Our work demonstrates that – contrary to widely accepted opinion – a continuous representation is not strictly necessary to allow fast and high-quality radiance field training.
Result The trade-off, broken in one table row
Gap / Idea There is no neural network in this pipeline
Method Fig. 1 — the quality/speed frontier, drawn
Question What is the "real-time" number actually conditioned on?
Method Eq. 1–3: NeRF and splatting already share one image formation model
Gap / Idea Why structured grids cap quality and speed at once
Reference Mip-NeRF360 [Barron et al. 2022] — the quality bar
Reference InstantNGP and Plenoxels — the speed bar
Reference Pulsar [Lassner and Zollhofer 2021] — borrowed, then corrected
Question Why can this method skip MVS when every prior point method needed it?
Method The primitive: a 3D Gaussian with full anisotropic covariance
Method $\Sigma = RSS^{T}R^{T}$: keeping covariance both valid and optimizable
Method Adaptive density control: clone what's missing, split what's smeared
Method Tile rasterizer: sort once per frame, not once per pixel
Method The loss: L1 plus D-SSIM, and nothing else
Claim Unlimited gradient depth complexity — the choice worth 11 dB
Memo The constants that look arbitrary: $\phi=1.6$, 0.0002, 100, 3000
Question When does per-tile (rather than per-pixel) sorting become visible?
Question Why does one gradient signal flag both under- and over-reconstruction?
Question Why recover opacities by division instead of storing per-pixel lists?
Result Ours-7K vs Ours-30K: two usable operating points
Result Ablations: which component is actually load-bearing
Result Fig. 5 — where the quality wins actually are
Memo The column the headline numbers skip: memory
Question How much of the comparison is protocol, and what does it not control?
Gap / Idea Limitations, and the shared root under most of them
Gap / Idea 80% of training is still Python — the speedup left on the table
Gap / Idea The open question the authors name: can Gaussians give you a mesh?
Reference Code, data and the interactive viewer are released