Learning Camera Control in Dynamic Scenes from Limited Demonstrations

R. Hanocka*, J. Assa, D. Cohen-Or, R. Giryes

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review


In this work, we present our strategy for camera control in dynamic scenes with multiple people (sports teams). We learn a generic model of the player dynamics offline in simulation. We use only a few sparse demonstrations of a user's camera control policy to learn a reward function to drive camera motion in an ongoing dynamic scene. Key to our approach is the creation of a low-dimensional representation of the scene dynamics which is independent of the environment action and rewards, which enables learning the reward function using only a small number of examples. We cast the user-specific control objective as an inverse reinforcement learning problem, aiming to learn an expert's intention from a small number of demonstrations. The learned reward function is used in combination with a visual model predictive controller (MPC). We learn a generic scene dynamics model that is agnostic to the user-specific reward, enabling reusing the same dynamics model for different camera control policies. We show the effectiveness of our method on simulated and real soccer matches.

Original languageEnglish
Pages (from-to)427-437
Number of pages11
JournalComputer Graphics Forum
Issue number1
StatePublished - Feb 2022


  • animation
  • control
  • motion planning


Dive into the research topics of 'Learning Camera Control in Dynamic Scenes from Limited Demonstrations'. Together they form a unique fingerprint.

Cite this