On Learning Mechanical Laws of Motion from Video Using Neural Networks

Pradyumna Chari1 Yunhao Ba1 Shijie Zhou1 Chinmay Talegaonkar12 Shreeram Athreya1 Achuta Kadambi13

University of California Los Angeles1 University of California, San Diego2 California Nanosystems Institute, UCLA3

image Figure 1. Recognizing physical equations from visual cues without human intervention. Here, we show how an input video of projectile motion can be processed by our method to recover both the governing equation of motion, as well as two governing parameters of initial velocities (both horizontal and vertical). These can be used for several applications in computer vision.

In computer vision, physics plays an important role in several applications. In this work, we teach a machine to detect the mechanical laws of motion of physical objects using video, and show how the results can be useful for computer vision tasks. We assume no prior knowledge of physics, beyond a temporal stream of bounding boxes. The problem is very difficult because a machine must learn not only a governing equation (e.g. projectile motion) but also the existence of governing parameters (e.g. velocities). We evaluate our ability to represent the physical laws of motion in video, such as the movement of a projectile or circular motion, in both real and constructed videos. These elementary tasks have textbook governing equations and enable ground truth verification of our approach. To establish the importance of the proposed method, we show a real-world use case in the domain of object tracking in confounding scenes, where existing state-of-the-art algorithms fail. Incorporating physics into computer vision not only serves the purpose of curiosity-driven research, but also provides an inductive bias for computer vision applications like object tracking.


  • Paper (Coming soon!)
  • Low-res paper (Coming soon!)
  • Code and data (Coming soon!)


Coming soon!



Achuta Kadambi
Assistant Professor
Electrical and Computer Engineering Department