A common theme in many extremal problems in graph theory is the relation between local and global properties of graphs. One of the most celebrated results of this type is the Ruzsa-Szemerédi triangle removal lemma, which states that if a graph is e-far from being triangle free, then most subsets of vertices of size C(e) are not triangle free. Unfortunately, the best known upper bound on C(e) is given by a tower-type function, and it is known that C(e) is not polynomial in e-1. The triangle removal lemma has been extended to many other graph properties, and for some of them the corresponding function C(e) is polynomial. This raised the natural question, posed by Goldreich in 2005 and more recently by Alon and Fox, of characterizing the properties for which one can prove removal lemmas with polynomial bounds. Our main results in this paper are new sufficient and necessary criteria for guaranteeing that a graph property admits a removal lemma with a polynomial bound. Although both are simple combinatorial criteria, they imply almost all prior positive and negative results of this type. Moreover, our new sufficient conditions allow us to obtain polynomially bounded removal lemmas for many properties for which the previously known bounds were of tower type. In particular, we show that every semialgebraic graph property admits a polynomially bounded removal lemma. This confirms a conjecture of Alon.