While the notion of social-ecological system resilience is widely accepted and applied, the issue of “resilience for whom” is clearly ignored. This phenomenon has also occurred in Taiwan. This article explores the roots of, and a possible solution to, this issue through a case study in the context of Taiwan’s indigenous peoples. The Danungdafu area, the focal social-ecological system, was studied. Qualitative research methods and an action-oriented research approach were employed. For a long period, the central government shaped the political, economic, social, institutional, and ecological contexts; dominated resilience discourses and determined the problem-framing and problem-solving agenda; defined the scale and levels at which social-ecological system governance issues were addressed; and determined the knowledge system used to define and solve problems. After 2011, a new participatory governance regime emerged. Multiple stakeholders, including indigenous communities, began to contribute to resilience discourses and influenced governance and trade-offs among differing governance goals. However, under the established structures dominated by Han people, indigenous views, rights, and well-being continue to be ignored. Affirmative action is required to recognize and safeguard indigenous rights. A practical institutional pathway is available to facilitate the transformation from “resilience for mainstream society” to “resilience for indigenous people” in indigenous territories.