Using the decomposition of rational conformal filed theory characters for the $\Z_k$ parafermion quantum Hall droplets for general $k=2,3,\dots$, we derive analytically the full modular $S$ matrix for these states, including the $\uu$ parts corresponding to the charged sector of the full conformal field theory and the neutral parafermion contributions corresponding to the diagonal affine coset models. This precise neutral-part parafermion $S$ matrix is derived from the explicit relations between the coset matrix and those for the numerator and denominator of the coset and the latter is expressed in compact form due to the level-rank duality between the affine Lie algebras $\widehat{\frak{su}(k)_2}$ and $\widehat{\frak{su}(2)_k}$. The exact results obtained for the $S$ matrix elements are expected to play an important role for identifying interference patterns of fractional quantum Hall states in Fabry-P\'erot interferometers which can be used to distinguish between Abelian and non-Abelian statistics of quasiparticles localized in the bulk of fractional quantum Hall droplets as well as for nondestructive interference measurement of Fibonacci anyons which can be used for universal topological quantum computation