The labyrinth screw pump is a new type of low-flow rotor pump with a simple structure and good sealing performance. It is suitable for the transport of high-viscosity, high-gas-content, and particle-containing media. In this study, a rectangular labyrinth screw pump was used as the research object. The effect of the medium viscosity on the performance of the labyrinth pump was studied through numerical simulations, and the correctness of the simulation method was verified using existing test data. The efficiency and head of the labyrinth screw pump were selected as the optimization objectives, and the pump structural parameters were selected as the optimization parameters. A structural optimization model of the labyrinth screw pump based on response surface theory was established. The structural parameters of the labyrinth pump were reasonably simplified through size correlations, and then parameter sensitivity analysis was performed to determine the important structural parameters that needed to be optimized. The OSFD (optimized space-filling design) was used to combine the optimized parameters and generate the sample space. The response surface theory was combined with a neural network prediction model and a multi-objective genetic algorithm to perform optimization calculations. The results showed that there was an interactive influence between the structural parameters of the stator and rotor of the labyrinth screw pump. Compared with the original model, the optimized model pump had an efficiency increase of 13.55% and a lift increase of 19.53% when conveying a medium with a viscosity of 133 cp.