Due to poor image quality as well as the difficulty of modeling the non-rigid heart motion, motion information has rarely been used in the past for angiogram analysis. In this paper we propose a new motion feature for the purpose of classifying angiogram videos according to their viewpoints. Specifically, local motion content of the video around the anatomical structures cardiac vessels is represented using the so-called "motion context", a motion histogram representation in polar coordinates within a local patch. The global motion layout is captured as pyramid histograms of the motion context (PHMC) in a manner similar to that proposed by the Spatial Pyramid Kernel . The PHMC is a robust representation of the motion features in a video sequence. Through experiments on a large database of angiograms obtained from both diseased and control subjects, we show that our technique consistently outperforms state-of-the-art methods in the angiogram classification test.