Breast cancer incidence has been rising steadily during the past few decades. It is the second leading cause of death in women. If it is diagnosed early, there is a good possibility of recovery. Mammography is proven to be an excellent screening technique for breast tumor diagnosis, but its detection and classification in mammograms remain a significant challenge. Previous studies’ major limitation is an increase in false positive ratio (FPR) and false negative ratio (FNR), as well as a drop in Matthews correlation coefficient (MCC) value. A model that can lower FPR and FNR while increasing MCC value is required. To overcome prior research limitations, a modified network of YOLOv5 is used in this study to detect and classify breast tumors. Our research is conducted using publicly available datasets Curated Breast Imaging Subset of DDSM (CBIS-DDSM). The first step is to perform preprocessing, which includes image enhancing techniques and the removal of pectoral muscles and labels. The dataset is then annotated, augmented, and divided into 60% for training, 30% for validation, and 10% for testing. The experiment is then performed using a batch size of 8, a learning rate of 0.01, a momentum of 0.843, and an epoch value of 300. To evaluate the performance of our proposed model, our proposed model is compared with YOLOv3 and faster RCNN. The results show that our proposed model performs better than YOLOv3 and faster RCNN with 96% mAP, 93.50% MCC value, 96.50% accuracy, 0.04 FPR, and 0.03 FNR value. The results show that our suggested model successfully identifies and classifies breast tumors while also overcoming previous research limitations by lowering the FPR and FNR and boosting the MCC value.