In the last two decades, the 60 years old view that in utero exposure to testosterone irreversibly masculinizes the brain of males away from a default female form has been replaced by a complex scenario according to which sex affects the brains of both females and males via multiple mechanisms, which are susceptible to internal and external factors. These observations led to the “mosaic” hypothesis—the expectation that the degree of “maleness”/“femaleness” of different features within a single brain would not be internally consistent. Following a short review of the animal studies providing the basis of the mosaic hypothesis, I describe three studies conducted in humans that assessed internal consistency in regional volume, cortical thickness, and connectivity as revealed by magnetic resonance imaging (MRI); in neuronal numbers in the postmortem hypothalamus; and in changes in regional volume and cortical thickness (assessed with MRI) following exposure to extreme real-life stress. The conclusion from these studies, that human brains are largely composed of unique mosaics of female-typical and male-typical features, was supported by recent findings that the brain “types” typical of women are also typical of men, and vice versa. Lastly, I discuss criticism of the mosaic hypothesis and suggest replacing the framework of a male–female continuum with thinking about mosaic brains residing in a multidimensional space.