The number of pollen grains varies within and between species. However, little is known about the molecular basis of this quantitative trait, in contrast with the many studies available on cell differentiation in the stamen. Recently, the first gene responsible for pollen number variation, REDUCED POLLEN NUMBER1 (RDP1), was isolated by genome-wide association studies of Arabidopsis thaliana and exhibited the signature of natural selection. This gene encodes a homolog of yeast Mrt4 (mRNA turnover4), which is an assembly factor of the large ribosomal subunit. However, no further data were available to link ribosome function to pollen development. Here, we characterized the RDP1 gene using the standard A. thaliana accession Col-0. The frameshift mutant, rdp1-3 generated by CRISPR/Cas9 revealed the pleiotropic effect of RDP1 in flowering, thus demonstrating that this gene is required for a broad range of processes other than pollen development. We found that the natural Col-0 allele conferred a reduced pollen number against the Bor-4 allele, as assessed using the quantitative complementation test, which is more sensitive than transgenic experiments. Together with a historical recombination event in Col-0, which was identified by sequence alignment, these results suggest that the coding sequence of RDP1 is the candidate region responsible for the natural phenotypic variation. To elucidate the biological processes in which RDP1 is involved, we conducted a transcriptome analysis. We found that genes responsible for ribosomal large subunit assembly/biogenesis were enriched among the differentially regulated genes, which supported the hypothesis that ribosome biogenesis is disturbed in the rdp1-3 mutant. Among the pollen-development genes, three key genes encoding basic helix-loop-helix (bHLH) transcription factors (ABORTED MICROSPORES (AMS), bHLH010, and bHLH089), as well as direct downstream genes of AMS, were downregulated in the rdp1-3 mutant. In summary, our results suggest a specialized function of ribosomes in pollen development through RDP1, which harbors natural variants under selection.