Hyperspectral technology has been used to identify pigments that adhere to the surfaces of polychrome artifacts. However, the colors are often produced by the mixing of pigments, which requires that the spectral characteristics of the pigment mixtures be considered before pigment unmixing is conducted. Therefore, we proposed an experimental approach to investigate the nonlinear degree of spectral reflectance, using several mixing models, and to evaluate their performances in the study of typical mineral pigments. First, five mineral pigments of azurite, malachite, cinnabar, orpiment, and calcite were selected to form five groups of samples, according to their different mass ratios. Second, a fully constrained least squares algorithm based on the linear model and three algorithms based on the nonlinear model were employed to calculate the proportion of each pigment in the mixtures. We evaluated the abundance accuracy as well as the similarity between the measured and reconstructed spectra produced by those mixing models. Third, we conducted pigment unmixing on a Chinese painting to verify the applicability of the nonlinear model. Fourth, continuum removal was also introduced to test the nonlinearity of mineral pigment mixing. Finally, the results indicated that the spectral mixing of different mineral pigments was more in line with the nonlinear mixing model. The spectral nonlinearity of mixed pigments was higher near to the wavelength corresponding to their colors. Meanwhile, the nonlinearity increased with the wavelength increases in the shortwave infrared bands.