Cardiometabolic risk factors (CMRs) are associated with accelerated brain aging and increased risk for sex-dimorphic illnesses such as Alzheimer’s Disease (AD). Yet, it is unknown how CMRs interact with sex and apolipoprotein E-ε4 (APOE4), a known genetic risk factor for AD, to influence brain age across different life stages. Using age prediction based on multi-shell diffusion-weighted imaging data in 21,308 UK Biobank participants, we investigated whether associations between white matter Brain Age Gap (BAG) and body mass index (BMI), waist-to-hip ratio (WHR), body fat percentage (BF%), and APOE4 status varied i) between males and females, ii) according to age at menopause in females, and iii) across different age groups in males and females. We report sex differences in associations between BAG and all three CMRs, with stronger positive associations among males com- pared to females. Higher BAG (older brain age relative to chronological age) was associated with greater BMI, WHR, and BF% in males, whereas in females, higher BAG was associated with greater WHR, but not BMI and BF%. These divergent associations were most prominent within the oldest group of females (66-81yrs), where higher BF% was linked to lower BAG (younger brain age relative to chronological age). Earlier menopause transition was associated with higher BAG, but no interactions were found with CMRs. APOE4 status was not significantly associated with BAG, and no significant interactions with CMRs were found. In conclusion, the findings point to sex- and age-specific associations between body fat composition and brain age. Incorporating sex as a factor of interest in studies addressing cardiometabolic risk may promote sex-specific precision medicine, consequently improving health care for both males and females.