Recent successes in Monte Carlo methods for simulating fermionic quantum impurity models have been based on diagrammatic resummation techniques, but they are restricted by the need to sum over factorially large classes of diagrams individually. We present a fast algorithm for summing over the diagrams appearing in Inchworm hybridization expansions. The method relies on the inclusion-exclusion principle to reduce the scaling from factorial to exponential. We analyze the growth rate and compare with related algorithms for expansions in the many-body interaction. An implementation demonstrates that for a simulation of a concrete physical model at reasonable parameters and accuracy within the Inchworm hybridization expansion, our algorithm not only scales better asymptotically, but also provides performance gains of approximately two orders of magnitude in practice over the previous state of the art.