Networks of semiflexible or stiff polymers such as most biopolymers are known to deform inhomogeneously when sheared. The effects of such nonaffine deformation have been shown to be much stronger than for flexible polymers. To date, our understanding of nonaffinity in such systems is limited to simulations or specific 2D models of athermal fibers. Here, we present an effective medium theory for nonaffine deformation of semiflexible polymer and fiber networks, which is general to both 2D and 3D and in both thermal and athermal limits. The predictions of this model are in good agreement with both prior computational and experimental results for linear elasticity. Moreover, the framework we introduce can be extended to address nonlinear elasticity and network dynamics.