The study of the single-particle erosion mechanism is essential to understand the material removal mechanism in the non-contact polishing process and ultimately ensure the high-efficiency, non-damage, and ultra-smooth processing of optical glass. In this study, the theoretical model of smoothed particle hydrodynamics (SPH) is established to reveal the dynamic removal process of a single particle impacting the optical glass. The single-particle erosion mechanisms, which include ductile–brittle transition, crack initiation, and propagation, are discussed in detail through theoretical simulation. A series of particle impact experiments are designed to validate the correctness of the SPH model. The experimental data show good agreement with the simulation results in terms of the depth and width of the eroded craters. Thereafter, the SPH simulation is conducted by studying the effect of various impact parameters, such as impact speed, impact angle, and abrasive diameter, on the material removal process. With the gradual increase of impact velocity and particle size, the material removal mode changes from plastic removal to brittle removal. Although the large impact velocity and particle size increase the material removal rate, they lead to the occurrence of brittle removal and reduce the surface and sub-surface quality. When the impact angle is between 45° and 75°, the material removal rate is the largest, and the increase of the material removal rate does not cause damage to the subsurface layer of the material.