From replication to substantiation: A complexity theory perspective

In contemporary methodological thinking, replication holds a central place. However, relatively little attention has been paid to replication in the context of complex dynamic systems theory (CDST), perhaps due to uncertainty regarding the epistemology–methodology match between these domains. In this paper, we explore the place of replication in relation to open systems and argue that three conditions must be in place for replication research to be effective: results interpretability, theoretical maturity, and terminological precision. We consider whether these conditions are part of the applied linguistics body of work, and then propose a more comprehensive framework centering on what we call substantiation research, only one aspect of which is replication. Using this framework, we discuss three approaches to dealing with replication from a CDST perspective theory. These approaches are moving from a representing to an intervening mindset, from a comprehensive theory to a mini-theory mindset, and from individual findings to a cumulative mindset.

Replication and Open Science in Applied Linguistics Research

This chapter addresses the role and place of replication research and open science practices in advancing theory building and new research directions in the field of applied linguistics. The chapter begins by describing what replication research is, what the most common types of replication study are, and why carrying out replication matters. Close attention is paid throughout to the ways in which replication benefits from and contributes to a variety of open science initiatives, including open materials, open access and preprints, and preregistration.

Replicating corpus-based research in English for academic purposes: Proposed replication of Cortes () and Biber and Gray ()

Accurate description of language use is central to English for academic purposes (EAP) practice. Thanks to the development of corpus tools, it has been possible to undertake systematic studies of language in academic contexts. This line of research aims to provide detailed and accurate characterization of academic communication and to ultimately inform EAP practice. Very few studies, however, have attempted to ascertain whether, and to what the extent, corpus-based findings have achieved such goals. The diverse nature of EAP, and the unique methodological challenges involved in compiling and using corpora, provide sufficient incentive for replication research in this area. The present article makes a case for replication of corpus-based studies in the field of EAP. It argues that replication research not only enhances the credibility of corpus linguistics for EAP pedagogy and research but also provides practical advice for EAP teachers and materials designers. It then looks at how two key corpus-based studies on the topic, Cortes (2013) and Biber and Gray (2010), can be replicated with respect to replication approaches described in Porte (2012).

Are business and management journals anti-replication? An analysis of editorial policies

Purpose This study aims to explore the degree to which the editorial policies of business and management journals explicitly or implicitly discourage replication studies. Design/methodology/approach The paper examines differences in editorial policy toward replication studies relative to journal quality, age and sub-discipline area. A total of 600 journals (listed as Q1 and Q2 in Scopus) were selected for the current study. Findings The results reveal that out of 600 selected journals, only 28 (4.7%) were explicitly open to considering replication studies, while 331 (55.2%) were neutral, being neither explicitly nor implicitly dismissive of replication studies. A further 238 (39.7%) were implicitly dismissive of replication studies, and the remaining 3 (0.5%) journals were explicitly disinterested in considering replication studies for publication. CiteScore and source normalized impact factor of neutral journals were significantly lower than those of journals, which were implicitly discouraging replication research. With regard to the journals implicitly discouraging replications (238), journals in the subcategory of business and international management (51) had the highest percentage (21.4%) followed by strategy and management 30 (12.6%) and OB and HR 25 (10.5%). Originality/value The available literature does not explore the degree to which the editorial policies of business and management journals explicitly or implicitly discourage replication studies. The current study attempts to address this gap in the literature. Given the lack of support for replications among business and management journals, the current paper sets forth the suggested steps which are deemed crucial for moving beyond the replication crisis in the business and management field.

Context Dependency as a Predictor of Replicability

We scrutinize the argument that unsuccessful replications—and heterogeneous effect sizes more generally—may reflect an underappreciated influence of context characteristics. Notably, while some of these context characteristics may be conceptually irrelevant (as they merely affect psychometric properties of the measured/manipulated variables), others are conceptually relevant as they qualify a theory. Here, we present a conceptual and analytical framework that allows researchers to empirically estimate the extent to which effect size heterogeneity is due to conceptually relevant versus irrelevant context characteristics. According to this framework, contextual characteristics are conceptually relevant when the observed heterogeneity of effect sizes cannot be attributed to psychometric properties. As an illustrative example, we demonstrate that the observed heterogeneity of the “moral typecasting” effect, which had been included in the ManyLabs 2 replication project, is more likely attributable to conceptually relevant rather than irrelevant context characteristics, which suggests that the psychological theory behind this effect may need to be specified. In general, we argue that context dependency should be taken more seriously and treated more carefully by replication research.

Addressing a Crisis of Generalizability with Large-Scale Construct Validation

Yarkoni describes a grim state of psychological science in which the gross misspecification of our models and specificity of our operationalizations produce claims with generality so narrow that no one would be interested in them. We consider this a generalizability of construct validity issue and discuss how construct validation research should precede large-scale replication research. We provide ideas for a path forward by suggesting psychologists take a few steps back. By retooling large-scale replication studies, psychologists can execute the descriptive research needed to assess the generalizability of constructs. We provide examples of reusing large-scale replication data to conduct construct validation research post hoc. We also discuss proof of concept research that is on-going at the Psychological Science Accelerator. Big team psychology makes large-scale construct validity and generalizability research feasible and worthwhile. We assert that no one needs to quit the field, in fact, there is plenty of work to do. The optimistic interpretation is that if psychologists focus less on generating new ideas and more on organizing, synthesizing, measuring, and assessing constructs from existing ideas, we can keep busy for at least 100 years.