Ubiquitous 3D Vision in the Wild

As cameras become ubiquitous, there is an increasing opportunity to reliably detect, reconstruct, and track in 3D the visual data in those footages for downstream applications, like surveillance, user intent understanding, or creative purposes. In this talk, we discuss our recent progress in 3D object scene understanding from three different platforms: a single infrastructure camera, a single wearable smart glasses, and collections of data captured by smartphone cameras. Firstly, we present Snipper, a novel framework to jointly detect, track, and forecast future human motion from an RGB video snippet. Despite its simplicity, Snipper outperforms many existing methods in stationary infrastructure camera tracking settings. Secondly, we present AriaHuman, the first large-scale 3D multi-human tracking benchmark acquired under different environments and activities from a smart glass setting, and a baseline method that takes advantage of the multiple cameras stream commonly available in the glass settings. Finally, we present BANMo, the first method to create 3D textured shape and estimate the motion of humans and animals in a unified manner from many casual videos that were not captured simultaneously.