Spatial appearance - part of the complete visual scene signal

Most cameras integrate incoming light into a lateral grid of intensity or colour values. Edges, textures and shapes support object detection and scene interpretation, yet the image is a compressed account of the optical field: the arrival time of each photon is discarded. Time-resolved imaging preserves part of that timing. Rather than recording only how much light arrives at a pixel, the sensor records when detected photons arrive relative to a known illumination event. The result may be a temporal histogram per zone or pixel, a sequence of time gates, or a stream of photon timestamps. Spatial appearance remains useful; it is accompanied by a measurement of light’s temporal behaviour.

From travel time to scene structure

The best-known use of timing is direct time-of-flight (dToF) ranging. For a short pulse that travels to a surface and returns, the round-trip delay estimates distance:

d = c ∆t / 2

where c is the speed of light and ∆t is the measured delay. Timing uncertainty σ_∆t propagates into range uncertainty approximately as σ_d ≈ (c/2) σ_∆t. In practice accuracy also depends on photon statistics, calibration, scene geometry and target reflectance.

Timing can contain more than a single distance estimate. Photons may be scattered, absorbed, reflected or transmitted along different paths. Their arrival-time distribution—the transient—therefore reflects the interaction between illumination, geometry, material and sensor response. This is why time-resolved systems appear in depth sensing, lidar and biophotonics. Reviews of single-photon avalanche diode (SPAD) imagers describe how photon-counting and time-stamping have enabled a wide range of such measurements [1,2].

Why the temporal channel may complement spatial appearance

Two objects can look similar under conventional RGB imaging while differing in surface finish, translucency or subsurface scattering. A temporal measurement may supply complementary evidence because those physical interactions reshape the distribution of detected photons. In principle, a vision system can use spatial appearance to localise an object and time-resolved data to refine a material or surface class. This is not a claim that every material has a unique, invariant timing signature. The measured response depends on wavelength, illumination geometry, distance, surface roughness, background light, detector response and the scene itself. Different materials can produce overlapping signals; the same material can look different under changed conditions. Temporal information is best treated as an additional measurement channel, not a universal material label.

Recent work on spatiotemporal fusion formalises a cross-feature sourcing principle: spatial features (layout, edges, category) are drawn from laterally resolved intensity maps, while material composition and range are derived from co-registered dToF transients rather than inferred from appearance alone [3]. That separation aligns the sensing pipeline with the space and time dimensions of optical measurement and reduces reliance on appearance-dominated shortcuts.

Sensors and algorithms must be designed together

The added information has a cost. Time-resolved sensors typically require pulsed illumination, precise timing electronics, photon-counting detectors, calibration and substantial data movement. SPAD arrays can detect individual photons and provide fine timing, but practical systems must account for dark counts, afterpulsing, dead time, pile-up, ambient light and finite photon budgets [1,2]. Higher temporal resolution does not automatically improve task performance if the signal is noisy or the relevant differences lie below the system’s resolvable scale.

A measured histogram can be modelled, in simplified form, as the convolution of the emitted pulse shape, the material impulse response and the instrument response function. Algorithms should respect that structure: a temporal histogram is a sampled distribution shaped by both scene physics and the instrument, not simply another colour channel. Models need evaluation for robustness across distances, illumination levels, sensor units and environments. Where timing data are fused with RGB images, registration errors can create misleading associations between an object and its temporal signature.

How should we test whether timing really helps?

A convincing evaluation should compare at least three conditions: RGB-only input, time-resolved input alone, and a fused model under the same data partitions. Splits should separate scenes, objects or acquisition sessions where possible, rather than allowing near-duplicate frames into both training and test sets. Researchers should report not only average accuracy but also uncertainty, calibration status, performance under low photon counts and failure rates for difficult classes. Ablation studies can reveal which parts of the temporal distribution matter: do early and late photons contribute different information? Does a compact set of temporal features perform as well as the full histogram? How much performance remains when ambient illumination changes? These questions help distinguish genuine physical complementarity from improvements caused by dataset bias or extra model capacity.

Empirical dual-module pipelines that co-register RGB intensity with SPAD dToF transients have reported combined spatial–material scores in the mid-to-high 90 % range on controlled household categories under pose and illumination sweeps, with material modules approaching 99 % validation accuracy at millisecond-class CPU inference [3]. Such numbers are informative only when accompanied by the protocol that produced them—sensor geometry, photon budget, split strategy and spoofing or unknown-class tests.

A useful direction for multimodal perception

Time-resolved imaging offers a broader lesson for machine perception: a camera’s output is determined not only by the scene, but also by what the sensor chooses to measure. Combining spatial appearance with photon timing may be useful when geometry or material interactions matter, but its value must be demonstrated under controlled and realistic conditions. The next step is not simply to collect more data. It is to design experiments that identify when temporal information changes a decision, quantify the cost of obtaining it, and establish where it fails. Such evidence would help determine whether time-resolved imaging is best suited to specialised scientific instruments, industrial inspection, robotics or more general-purpose vision systems.

References

[1] Bruschini, C. et al. “Single-photon avalanche diode imagers in biophotonics: review and outlook.” Light: Science & Applications 8, 87 (2019). https://doi.org/10.1038/s41377-019-0191-5

[2] Piron, F. et al. “A review of single-photon avalanche diode Time-of-Flight imaging sensor arrays.” IEEE Sensors Journal 21(11), 12654–12666 (2021). https://doi.org/10.1109/JSEN.2020.3039362

[3] Minka, D. A. “Beyond Appearance: Intelligent Spatiotemporal Vision for Material-Aware Object Detection.” Research Square preprint (2026). https://doi.org/10.21203/rs.3.rs-10412364/v1