We study the structure of non-expanding sets in the Grassmann graph. We put forth a hypothesis stating that every small set whose expansion is smaller than 1–δ must be correlated with one of a specified list of sets which are isomorphic to smaller Grassmann graphs. We develop a framework of Fourier analysis for analyzing functions over the Grassmann graph, and prove that our hypothesis holds for all sets whose expansion is below 3/4. Our work is motivated by [DKK+18], wherein the authors show that a linearity agreement hypothesis implies an NP-hardness gap of 1/2–ε vs. ε for Unique Games and other inapproximability results. Barak, Kothari and Steurer show that the hypothesis in this work implies the linearity agreement hypothesis [DKK+18]. Following initial publication of this work, our hypothesis was proved in [KMS18].