Cultural transmission, competition for prey, and the evolution of cooperative hunting

Although cooperative hunting is widespread among animals, its benefits are unclear. At low frequencies, cooperative hunting may allow predators to escape competition and access bigger prey that could not be caught by a lone cooperative predator. Cooperative hunting is a more successful strategy when it is common, but its spread can result in overhunting big prey, which may have a lower per-capita growth rate than small prey. We construct a one-predator species, two-prey species model in which predators either learn to hunt small prey alone or learn to hunt big prey cooperatively. Predators first learn vertically from parents, then horizontally (i.e. socially) from random individuals or siblings. After horizontal transmission, they hunt with their learning partner if both are cooperative, and otherwise they hunt alone. Cooperative hunting cannot evolve when initially rare unless predators (a) interact with siblings, or (b) horizontally transmit the cooperative behavior to potential hunting partners. Whereas competition for small prey favors cooperative hunting when this cooperation is initially rare, the frequency of cooperative hunting cannot reach 100% unless big prey is abundant. Furthermore, a mutant that increases horizontal learning can invade if cooperative hunting is present, but not at 100%, because horizontal learning allows pairs of predators to have the same strategy. Our results reveal that the interactions between prey availability, social learning, and degree of cooperation among predators may have important effects on ecosystems.