SIGGRAPH 2026 · ACM Transactions on Graphics

GenPIE
A Time-Resolved Plenoptic Imager

Recovering physically grounded, time-resolved plenoptic light transport from sparse real-world transient measurements.

Ziheng Wang¹, Siyuan Shen², Huanyu Xu², Kaichun Qiao^2,3, Longwen Zhang^2,3, Qixuan Zhang^2,3,
Qilin Sun^1,4, Shiying Li^2*, Jingyi Yu^2*

¹The Chinese University of Hong Kong, Shenzhen · ²ShanghaiTech University · ³Deemos Technology · ⁴Point Spread Technology
^*Corresponding authors

Paper Supplementary Video Code Data

GenPIE captures configurable views and illuminations, records transient light transport, and recovers plenoptic information for multi-bounce decomposition, time-resolved relighting, and time unwarping.

Abstract

From sparse photons to dense light transport.

GenPIE bridges sparse physical observations and high-dimensional plenoptic light transport by combining a reconfigurable transient imaging system, generative 3D priors, and physically based inverse transient rendering.

Capturing the full plenoptic light transport across spatial, angular, and temporal dimensions has long been a pursuit in computational imaging, yet it remains fundamentally constrained by the high dimensionality of the sampling space and the physical inaccessibility of scene regions due to self-occlusions. While time-resolved imaging records the temporal axis, existing methods are bottlenecked by the combinatorial complexity of the plenoptic function. This high dimensionality makes dense omni-dimensional sampling physically prohibitive. Simultaneously, tight coupling between illumination and viewpoint in current systems also precludes the full acquisition of plenoptic light transport. We present GenPIE, a Generative Plenoptic Imager designed to bridge the gap between sparse physical observations and high-dimensional light transport. We demonstrate that GenPIE supports a range of applications that are challenging for steady-state or purely neural methods, including disentangling multi-bounce light transport directly from captured transient videos, time unwarping, and time-resolved relighting.

Decoupled transient capture

Separate illumination and detection paths allow the system to probe indirect transport under varying views and lighting conditions.

Generative initialization

A 3D foundation model turns sparse transient-derived point clouds into complete geometric priors for otherwise under-observed regions.

Physical refinement

Differentiable transient path tracing and neural compensation align recovered geometry, materials, and illumination with real measurements.

Method

A physically grounded generative imager.

The pipeline first captures sparse but metrically meaningful geometry, then expands it with learned 3D priors, and finally enforces transient light-transport consistency through differentiable simulation.

Overview of the GenPIE framework. — **Framework overview.** GenPIE uses the dual-mode imaging system to capture direct time-of-flight geometry and plenoptic transient measurements. A 3D foundation model converts sparse dToF point clouds into an initial geometry, which is then explicitly optimized together with material appearance and illumination using differentiable path tracing grounded in the transient rendering equation.

Capture

Dual-mode imaging

Confocal mode provides direct time-of-flight geometry; decoupled mode captures plenoptic transients under configurable lighting.

Generate

3D prior

A foundation model produces a complete initial mesh from sparse point clouds and integrated intensity observations.

Optimize

Transient rendering

Geometry, material, and illumination are refined by matching rendered and measured transient videos.

Compensate

System response

A spatially aware temporal kernel models real SPAD jitter, optical aberrations, and calibration residuals.

Results

Recovered light transport becomes editable.

Because GenPIE recovers explicit, physically meaningful scene parameters, the result is not only a reconstruction: it can be decomposed, relit, temporally reinterpreted, and visualized through peak-time propagation.

Color reference scenes

Captured color images for the three scenes used throughout the result section.

Scene

Bunny

Scene

Statue

Scene

Sphere

Multi-bounce decomposition

Separated direct and higher-order indirect transport.

Case 1

Bunny

Case 2

Metal sphere I

Case 3

Metal sphere II

Time-resolved relighting

Relighting with virtual light sources and projected patterns.

Multiple Point Lights

Bunny

Multiple Point Lights

Sphere

SIGGRAPH Logo

Statue logo

SIGGRAPH Logo

Sphere logo

Time unwarping

Each pair compares the captured transient before and after unwarping. Bunny pairs stay 1:1; the cola bottle pair is cropped to the lower region at 2.54:1.