Seeing through Occlusion: Uncertainty-aware Joint Physical Tracking and Prediction

Arijit Dasgupta¹, Andrew D. Bolton², Vikash K. Mansinghka¹, Joshua B. Tenenbaum¹, Kevin A. Smith¹

¹Massachusetts Institute of Technology
²CHI FRO

Paper Code arXiv (coming soon)

Marquee Figure

Participants continuously anticipated whether a moving blue ball would hit Red or Green first in a 2.5D environment with gray occluders and black barriers. Our model JTAP integrates perception with physical reasoning through probabilistic inference, generating belief states with positional estimates (black dots), speed and direction estimates, and predictive trajectories (yellow lines) that capture the uncertainty in human judgments during occlusion despite the absence of changing visual evidence.

Abstract

Humans can track objects and predict their motion even when they are temporarily occluded. How does the absence of changing visual evidence alter predictive beliefs about a moving object? In our study, participants were tasked with continuously anticipating the destination of a simulated ball in occluded and un-occluded 2.5D environments. Our findings reveal that humans actively update their judgments throughout the period of occlusion while making predictions grounded in physical realism, even as occlusion impairs accuracy. To model this behavior, we integrate perception with physical reasoning, unifying tracking and prediction. This is implemented via massively parallel probabilistic inference in a hierarchical generative model for the motion of intermittently visible objects, represented using the GenJAX probabilistic programming platform. This model predicts time-varying human judgments more accurately than alternative models, suggesting that humans integrate perception and physics to reason about occluded motion.

Modeling and Inference

The probabilistic generative process in JTAP

The probabilistic generative process in JTAP, where states dynamically evolve over time, and observations are conditionally generated from the state. Predictions at each time-step evolves from current state 𝑘 time-steps ahead.

Parallel compute graph of massively parallel Sequential Monte Carlo (SMC) inference in JTAP

Parallel compute graph of massively parallel Sequential Monte Carlo (SMC) inference in JTAP. Future 𝑘 time-steps evolve from current inferred physical state. Implemented in GenJAX, a probabilistic programming framework in JAX.

Uncertainty in Occlusion

With Occlusion

Ball tracking with occlusion

Without Occlusion

Ball tracking without occlusion

JTAP demonstrates how uncertainty in mental representations evolves differently when tracking visible versus occluded objects. During occlusion, uncertainty about the object's physical state grows over time, leading to increased prediction uncertainty (visualized by the changing proportions of red and green beliefs). In contrast, continuously visible objects maintain lower uncertainty levels. When an occluded object reappears, uncertainty immediately resolves before becoming hidden again.

Cognitive Comparisons

Alternative hypotheses

Alternative hypotheses for joint tracking and prediction. The Frozen hypothesis keeps the same static beliefs during occlusion, while the Decaying hypothesis decays beliefs over time in that same period.

Illustrative examples

Examples showing how JTAP compares to alternative models and humans. The absence of changing visual evidence is evidence itself - as shown in Trial D, where humans gradually predict the red outcome when the ball fails to appear on the left side where it would have hit green. JTAP captures this reasoning pattern while alternative models cannot, as they don't update beliefs during occlusion.

BibTeX

Coming Soon

Acknowledgements

This work was supported in part by CoCoSys, one of seven centers in JUMP 2.0, a Semiconductor Research Corporation (SRC) program sponsored by DARPA, and by NSF grant 2121009.