Brittle crystals, such as coloured gems, have long been known to cleave with atomically smooth fracture surfaces, despite being impurity laden, suggesting that isolated atomic impurities do not generally cause cracks to deflect. Whether cracks can ever deviate when hitting an atomic defect, and if so how they can go straight in real brittle crystals, which always contain many such defects, is still an open question. Here we carry out multiscale molecular dynamics simulations and high-resolution experiments on boron-doped silicon, revealing that cracks can be deflected by individual boron atoms. The process, however, requires a characteristic minimum time, which must be less than the time spent by the crack front at the impurity site. Deflection therefore occurs at low crack speeds, leading to surface ridges which intensify when the boron-dopage level is increased, whereas fast-moving cracks are dynamically steered away from being deflected, yielding smooth cleavage surfaces.