Based on the principle of Lagrange mechanics, especially considering the effects of rotation damping and nonsynchronous damping, a radial 4-dimensional dynamic model of the flywheel bearing rotor system is proposed. Applying the Laplace eigenvalue method, the stability effects of rotational damping, nonsynchronous damping, and their coupling effects are investigated by means of root locus method. Under the control of the linear quadratic regulator, dynamical characteristics of the flywheel bearing rotor system with varied rotational damping and nonsynchronous damping are also studied. The results show that the rotation damping, nonsynchronous damping, and their coupling effects have vast and complex instability effects on high-speed flywheel bearing rotor system. However, there are three exceptions. The tiny proportional rotational damping, remaining below 12%, and the minuscule proportional co-nonsynchronous damping; the product of the nonsynchronous damping and the speed ratio below 5% both can enhance the stability of the system. Furthermore, in the situation that the counter-nonsynchronous damping is coupled with the large proportion of rotational damping, the stability of the system can also be boosted distinctly. On the other hand, the numerical experimental results show that the rotational damping and nonsynchronous damping have a beneficial effect on the flywheel system controlled by linear quadratic regulator. In addition, under the control of linear quadratic regulator, the transient dynamical behavior of the flywheel rotor system with rotational damping or co-nonsynchronous damping performed better than the flywheel rotor system with the coupled damping. The numerical simulations of the transient response of the flywheel rotor system under active control are consistent with some of the derived stability analysis results. The results about the stability analysis and the performance in vibration control give the suggestions for the instability control and fault detection of the system.