Evaluation as a Systemic Component of Organizational Change Efforts

1978 ◽  
Vol 22 (1) ◽  
pp. 404-408 ◽  
Author(s):  
Kerry A. Bunker
Keyword(s):  
Organizational Change ◽  
External Validity ◽  
Internal Validity ◽  
Organizational Training ◽  
Feedback Systems ◽  
Systems Model ◽  
Potential Benefits ◽  
Validity Research ◽  
Systemic Component ◽  
Change Efforts

A cyclical-feedback systems model is proposed as a format for structuring and utilizing the results of evaluative investigations of organizational change efforts. The issues of internal validity, external validity, research design and results utilization are discussed; and an organizational training evaluation example is presented to demonstrate the feasibility, practicality and potential benefits of the approach.

scholarly journals A novel preference-informed complementary trial (PICT) design for clinical trial research influenced by strong patient preferences

Trials ◽  
2021 ◽  
Vol 22 (1) ◽  
Author(s):  
Samina Ali ◽  
◽  
Gareth Hopkin ◽  
Naveen Poonai ◽  
Lawrence Richer ◽  
...  
Keyword(s):  
External Validity ◽  
Treatment Options ◽  
Internal Validity ◽  
Clinical Effectiveness ◽  
Drop Out ◽  
Innovative Design ◽  
Clinical Trial Research ◽  
Research Questions ◽  
Internal And External Validity ◽  
Treatment Population

Abstract Background Patients and their families often have preferences for medical care that relate to wider considerations beyond the clinical effectiveness of the proposed interventions. Traditionally, these preferences have not been adequately considered in research. Research questions where patients and families have strong preferences may not be appropriate for traditional randomized controlled trials (RCTs) due to threats to internal and external validity, as there may be high levels of drop-out and non-adherence or recruitment of a sample that is not representative of the treatment population. Several preference-informed designs have been developed to address problems with traditional RCTs, but these designs have their own limitations and may not be suitable for many research questions where strong preferences and opinions are present. Methods In this paper, we propose a novel and innovative preference-informed complementary trial (PICT) design which addresses key weaknesses with both traditional RCTs and available preference-informed designs. In the PICT design, complementary trials would be operated within a single study, and patients and/or families would be given the opportunity to choose between a trial with all treatment options available and a trial with treatment options that exclude the option which is subject to strong preferences. This approach would allow those with strong preferences to take part in research and would improve external validity through recruiting more representative populations and internal validity. Here we discuss the strengths and limitations of the PICT design and considerations for analysis and present a motivating example for the design based on the use of opioids for pain management for children with musculoskeletal injuries. Conclusions PICTs provide a novel and innovative design for clinical trials with more than two arms, which can address problems with existing preference-informed trial designs and enhance the ability of researchers to reflect shared decision-making in research as well as improving the validity of trials of topics with strong preferences.

Investigating the Youth Physical Activity Promotion Model: Internal Structure and External Validity Evidence for a Potential Measurement Model

2007 ◽  
Vol 19 (4) ◽  
pp. 420-435 ◽  
Author(s):  
David A. Rowe ◽  
Thomas D. Raedeke ◽  
Lenny D. Wiersma ◽  
Matthew T. Mahar
Keyword(s):  
Physical Activity ◽  
External Validity ◽  
Internal Validity ◽  
Physical Activity Promotion ◽  
Measurement Model ◽  
Measurement Properties ◽  
Validity Evidence ◽  
Potential Measurement ◽  
Activity Promotion ◽  
Youth Physical Activity

The purpose of the study was to investigate the measurement properties of questionnaires associated with the Youth Physical Activity Promotion (YPAP) model. Data were collected from 296 children in Grades 5–8 using several existing questionnaires corresponding to YPAP model components, a physical activity questionnaire, and 6 consecutive days of pedometer data. Internal validity of the questionnaires was tested using confirmatory factor analyses, and external validity was investigated via correlations with physical activity and body composition. Initial model fit of the questionnaires ranged from poor to very good. After item removal, all scales demonstrated good fit. Correlations with percentage body fat and objectively measured physical activity were low but in the theoretically predicted direction. The current study provides good internal validity evidence and acceptable external validity evidence for a brief set of questionnaire items to investigate the theoretical basis for the YPAP model.

scholarly journals Toward Experimental International Business

2015 ◽  
Vol 22 (4) ◽  
pp. 530-544 ◽  
Author(s):  
Arjen van Witteloostuijn
Keyword(s):  
International Business ◽  
External Validity ◽  
Design Methodology ◽  
Internal Validity ◽  
Cross Cultural ◽  
Web Based ◽  
Content Type ◽  
Experimental Laboratory ◽  
Experimental Management ◽  
Missed Opportunity

Purpose – The purpose of this paper is to argue that the time is ripe to establish a powerful tradition in Experimental International Business (IB). Probably due to what the Arjen van Witteloostuijn refers to as the external validity myth, experimental laboratory designs are underutilized in IB, which implies that the internal validity miracle of randomized experimentation goes largely unnoticed in this domain of the broader management discipline. Design/methodology/approach – In the following pages, the author explains why the author believes this implies a missed opportunity, providing arguments and examples along the way. Findings – Although an Experimental Management tradition has never really gained momentum, to the author, the lab experimental design has a very bright future in IB (and management at large). To facilitate the development of an Experimental IB tradition, initiating web-based tools would be highly instrumental. This will not only boost further progress in IB research, but will also increase the effectiveness and playfulness of IB teaching. Originality/value – Given the high potential of an Experimental IB, the Cross-Cultural and Strategic Management journal will offer a platform for such exciting and intriguing laboratory work, cumulatively contributing to the establishment of an Experimental IB tradition.

scholarly journals Designing Research With Qualitative Comparative Analysis (QCA): Approaches, Challenges, and Tools

2017 ◽  
Vol 49 (2) ◽  
pp. 356-386 ◽  
Author(s):  
Eva Thomann ◽  
Martino Maggetti
Keyword(s):  
Comparative Analysis ◽  
External Validity ◽  
Qualitative Comparative Analysis ◽  
Internal Validity ◽  
Specific Research ◽  
Extant Literature ◽  
Specific Mode ◽  
Research Context ◽  
Trade Offs ◽  
Conducting Research

Recent years have witnessed a host of innovations for conducting research with qualitative comparative analysis (QCA). Concurrently, important issues surrounding its uses have been highlighted. In this article, we seek to help users design QCA studies. We argue that establishing inference with QCA involves three intertwined design components: first, clarifying the question of external validity; second, ensuring internal validity; and third, explicitly adopting a specific mode of reasoning. We identify several emerging approaches to QCA rather than just one. Some approaches emphasize case knowledge, while others are condition oriented. Approaches emphasize either substantively interpretable or redundancy-free explanations, and some designs apply an inductive/explorative mode of reasoning, while others integrate deductive elements. Based on extant literature, we discuss issues surrounding inference with QCA and the tools available under different approaches to address these issues. We specify trade-offs and the importance of doing justice to the nature and goals of QCA in a specific research context.

scholarly journals High Fidelity Microsurgical Simulation: The Thiel Model and Evaluation Instrument

2018 ◽  
Vol 27 (2) ◽  
pp. 118-124 ◽  
Author(s):  
Andrei Odobescu ◽  
Isak Goodwin ◽  
Djamal Berbiche ◽  
Joseph BouMerhi ◽  
Patrick G. Harris ◽  
...  
Keyword(s):  
External Validity ◽  
Internal Validity ◽  
High Fidelity ◽  
Evaluation Instrument ◽  
Intra Class Correlation ◽  
Validity Measures ◽  
The Difference ◽  
Fidelity Model ◽  
Junior Residents ◽  
Internal And External Validity

Background: The Thiel embalmment method has recently been used in a number of medical simulation fields. The authors investigate the use of Thiel vessels as a high fidelity model for microvascular simulation and propose a new checklist-based evaluation instrument for microsurgical training. Methods: Thirteen residents and 2 attending microsurgeons performed video recorded microvascular anastomoses on Thiel embalmed arteries that were evaluated using a new evaluation instrument (Microvascular Evaluation Scale) by 4 fellowship trained microsurgeons. The internal validity was assessed using the Cronbach coefficient. The external validity was verified using regression models. Results: The reliability assessment revealed an excellent intra-class correlation of 0.89. When comparing scores obtained by participants from different levels of training, attending surgeons and senior residents (Post Graduate Year [PGY] 4-5) scored significantly better than junior residents (PGY 1-3). The difference between senior residents and attending surgeons was not significant. When considering microsurgical experience, the differences were significant between the advanced group and the minimal and moderate experience groups. The differences between minimal and moderate experience groups were not significant. Based on the data obtained, a score of 8 would translate into a level of microsurgical competence appropriate for clinical microsurgery. Conclusions: Thiel cadaveric vessels are a high fidelity model for microsurgical simulation. Excellent internal and external validity measures were obtained using the Microvascular Evaluation Scale (MVES).

External Validity: The Next Step for Systematic Reviews?

2016 ◽  
Vol 41 (4) ◽  
pp. 283-325 ◽  
Author(s):  
Sarah A. Avellar ◽  
Jaime Thomas ◽  
Rebecca Kleinman ◽  
Emily Sama-Miller ◽  
Sara E. Woodruff ◽  
...  
Keyword(s):  
Systematic Review ◽  
Systematic Reviews ◽  
External Validity ◽  
Home Visiting ◽  
Evaluation Research ◽  
Internal Validity ◽  
Reporting Guidelines ◽  
Sample Characteristics ◽  
Statistical Representativeness

Background: Systematic reviews—which identify, assess, and summarize existing research—are usually designed to determine whether research shows that an intervention has evidence of effectiveness, rather than whether an intervention will work under different circumstances. The reviews typically focus on the internal validity of the research and do not consistently incorporate information on external validity into their conclusions. Objectives: In this article, we focus on how systematic reviews address external validity. Methods: We conducted a brief scan of 19 systematic reviews and a more in-depth examination of information presented in a systematic review of home visiting research. Results: We found that many reviews do not provide information on generalizability, such as statistical representativeness, but focus on factors likely to increase heterogeneity (e.g., numbers of studies or settings) and report on context. The latter may help users decide whether the research characteristics—such as sample demographics or settings—are similar to their own. However, we found that differences in reporting, such as which variables are included and how they are measured, make it difficult to summarize across studies or make basic determinations of sample characteristics, such as whether the majority of a sample was unemployed or married. Conclusion: Evaluation research and systematic reviews would benefit from reporting guidelines for external validity to ensure that key information is reported across studies.

scholarly journals Beyond Power Struggles: A Multilevel Perspective on Incongruences at the Interface of Practice, Knowledge, and Identity in Radical Organizational Change

2018 ◽  
Vol 55 (1) ◽  
pp. 5-26 ◽  
Author(s):  
Barbara Kump
Keyword(s):  
Organizational Change ◽  
Radical Change ◽  
Individual Characteristics ◽  
Knowledge Gaps ◽  
Power Struggles ◽  
Organizational Goals ◽  
Fine Grained ◽  
Multilevel Perspective ◽  
Practice Knowledge ◽  
Change Efforts

Previous approaches to describing challenges inherent in radical organizational change have mainly focused on power struggles. A complementary but less researched view proposes that many problems occur because radical change causes certain incongruences within an organization. In line with the latter perspective, this article suggests that radical change leads to incongruences between “what they do” (practice), “what they know” (knowledge), and “who they are” (identity) as an organization; to achieve the change, these incongruences need to be accommodated by the organization’s individual members. The article takes a multilevel perspective and describes how in radical change organizational goals may interfere with individual characteristics at the intersections of practice, knowledge, and identity. This enables a fine-grained analysis of reasons why radical change efforts may fail, beyond power struggles. The model is concrete enough to help change managers foresee many practical problems, such as member disidentification, routine breakdowns, or knowledge gaps.

External Validity Reconsidered

2011 ◽  
Author(s):  
Diana C. Mutz
Keyword(s):  
Political Science ◽  
Real World ◽  
External Validity ◽  
Observational Studies ◽  
Internal Validity ◽  
Conventional Wisdom ◽  
Observational Research ◽  
Convincing Case

This chapter talks about the significance of generalizability. Experimentalists often go to great lengths to argue that student or other convenience samples are not problematic in terms of external validity. Likewise, a convincing case for causality is often elusive with observational research, no matter how stridently one might argue to the contrary. The conventional wisdom is that experiments are widely valued for their internal validity, and experiments lack external validity. These assumptions are so widespread as to go without question in most disciplines, particularly those emphasizing external validity, such as political science and sociology. But observational studies, such as surveys, are still supposed to be better for purposes of maximizing external validity because this method allows studying people in real world settings.

Sign in / Sign up

Export Citation Format

Share Document