Background: The rapid growth of the elderly population poses a huge challenge for people to access medical services. The key to get rid of the dilemma is for patients to go firstly to primary medical institutions. Existing studies have identified numerous factors that can affect patients’ health institution choice. However, we currently know little about the role of Internet use in the patients’ medical decisions. The objective of this study is to explore health-seeking behavior and institution choice under the background of the Internet era from the perspective of older adults, and to analyze whether the Internet could guide patients to the appropriate medical institution so as to accomplish hierarchical treatment. Methods: The dataset comprises 9416 people aged 45 or above from the China Health and Retirement Longitudinal Survey (CHARLS), which, through multistage cluster sampling, was conducted in 2011, 2013, and 2015. Logistic regression, PSM, and FE model are used to estimate the influence of Internet use on the health care decision-making behavior. Results: Internet use has a significant positive impact on the self-treatment of common diseases (β = 0.05, p < 0.05). In terms of medical institution choices, those who use Internet are more inclined to choose top-level hospitals than community health service institutions to treat common diseases (β = 0.06, p < 0.01). Conclusions: The Internet has lowered the obstacles to learning about common ailments, resulting in a substitution impact of self-treatment for hospital care. However, Internet use may aggravate older adults’ perception of the risk of disease, which exacerbates the tendency of going to higher-level medical institutions for medical treatment. The finding of the study is useful for further rational planning and utilization of the Internet in order to guide patients to appropriate medical institution, which helps to improve the efficiency of the overall medical and health services.