The quality of palm oil depends on the maturity level of the oil palm fresh fruit bunch (FFB). This research applied an optical spectrometer to collect the reflectance data of 96 FFB from unripe, ripe, and overripe classes for the maturity level classification. The spectrometer scanned the FFB from different parts, including apical, front equatorial, front basil, back equatorial, and back basil. Principal component analysis was carried out to extract principal components from the reflectance data of each of the parts. The extracted principal components were used in an ANOVA test, which found that the reflectance data of the front equatorial showed statistically significant differences between the three maturity groups. Then, the collected reflectance data was subjected to machine learning training and testing by using the K-Nearest Neighbor (KNN) and Support Vector Machine (SVM). The front equatorial achieved the highest accuracy, of 90.6%, by using SVM as classifiers; thus, it was proven to be the most optimal part of FFB that can be utilized for maturity classification. Next, the front equatorial dataset was divided into UV (180–400 nm), blue (450–490 nm), green (500–570 nm), red (630–700 nm), and NIR (800–1100 nm) regions for classification testing. The UV bands showed a 91.7% accuracy. After this, representative bands of 365, 460, 523, 590, 623, 660, 735, and 850 nm were extracted from the front equatorial dataset for further classification testing. The 660 nm band achieved an 89.6% accuracy using KNN as a classifier. Composite models were built from the representative bands. The combination of 365, 460, 735, and 850 nm had the highest accuracy in this research, which was 93.8% with the use of SVM. In conclusion, these research findings showed that the front equatorial has the better ability for maturity classification, whereas the composite model with only four bands has the best accuracy. These findings are useful to the industry for future oil palm FFB classification research.