Background: Colorectal cancer is associated with lifestyle characteristics such as diet, physical inactivity, obesity, and smoking, but these are not incorporated in screening recommendations. Moreover, the joint association of these factors with various colorectal polyps is not established. Methods: A case–control study, among consecutive subjects aged 40–70 years, undergoing colonoscopy. Cases with colorectal polyps were compared with controls. Detailed information was gathered regarding polyp histology and anatomic location, demographics, medical history, anthropometrics, and lifestyle. The healthy lifestyle index was estimated as the sum of: non-smoking, maintaining a healthy weight, healthy diet, and physical activity. Results: A total of 788 participants were included (cases n = 403, controls n = 385). The healthy lifestyle index had a negative association with colorectal polyps (OR = 0.72, 95% CI 0.62–0.85, P < 0.001), both adenomas and serrated polyps (OR = 0.75, 0.64–0.89, and OR = 0.59, 0.44–0.79, respectively), and both proximal and distal adenomas (OR = 0.77, 0.62–0.95, and OR = 0.73, 0.59–0.90, respectively). Adherence to ≥ 2 healthy lifestyle components was strongly related with colorectal polyps (OR = 0.50, 0.34–0.75, P = 0.001). Abstinence from smoking, and a healthy diet were the factors most strongly associated with lower odds of colorectal polyps (OR = 0.58, 0.42–0.79, and OR = 0.61, 0.44–0.85, respectively). Conclusions: Adherence to a healthy lifestyle (≥2 healthy lifestyle components) is inversely associated with colorectal polyps, especially serrated and distal polyps, with no dose–response association. Components most strongly associated with lower odds of colorectal polyps were maintaining a healthy diet and abstinence from smoking. Lifestyle-related characteristics may assist in risk stratification and are potential goals for colorectal neoplasia prevention.