Talking to animals is a fundamental human desire. The emergence of powerful AI algorithms, and specifically Large Language Models, has driven many to suggest that we are on the verge of fulfilling this wish. A few large scientific consortia have been formed around this topic and several commercial entities even offer such services. We frame the task of communicating with animals as ‘The Doctor Dolittle challenge’ and identify three main obstacles on the route to doing so. First, although generative AI models can create novel animal communication samples, it is very difficult to determine their context, and we will forever be biased by our human umwelt when doing so. Second, using AI to extract context in an unsupervised manner must be validated through controlled experiments aiming to measure the animals’ response. This is difficult, and moreover, AI algorithms tend to cling on to any available information and are thus prone to finding spurious correlations. And third, animal communication focuses on a restricted set of contexts, such as alarm and courtship, highly limiting our ability to communicate regarding other contexts. Nevertheless, using the tremendous power of novel AI methods to decipher and mimic animal communication is both fascinating and important. We thus define the criteria for passing the Doctor Dolittle challenge and call upon scientists to take on the mission.