scholarly journals Assessing the reporting and interpretation of non-significant results in the study of cognitive development: a systematic review

2021 ◽  
Author(s):  
Edward William Legg ◽  
Benjamin George Farrar ◽  
Aleksandra Lazić ◽  
Maleen Thiele ◽  
Dora Kampis ◽  
...  
Keyword(s):  
Systematic Review ◽  
Cognitive Development ◽  
Null Hypothesis ◽  
Empirical Studies ◽  
Statistical Procedure ◽  
Significance Testing ◽  
Development Research ◽  
Null Hypothesis Significance Testing ◽  
Time Points ◽  
Extract Information

Null Hypothesis Significance Testing is a statistical procedure widely used in cognitive development research. There is widespread concern that the results of this statistical procedure are misinterpreted and lead to unsubstantiated claims about studies’ outcomes. Two particularly pertinent issues for research on cognitive development are: i) treating a non-significant result as evidence of no difference or no effect, and ii) treating a non-significant result in one group/condition and a significant result in another as evidence of a difference between groups/conditions. The current study focuses on quantifying the extent to which these two issues can be observed in the published literature on cognitive development. To this end, we will systematically search for empirical studies investigating cognitive development in 0-to-16-year-old children that have been published at two time points, namely in 1999 and 2019. For each of the two issues, we will extract information from 300 published articles, 150 per publication year.

scholarly journals A Frequentist Alternative to Significance Testing, p-Values, and Confidence Intervals

Econometrics ◽  
2019 ◽  
Vol 7 (2) ◽  
pp. 26 ◽  
Author(s):  
David Trafimow
Keyword(s):  
Present Article ◽  
Confidence Intervals ◽  
Null Hypothesis ◽  
A Priori ◽  
Significance Testing ◽  
Population Parameters ◽  
Null Hypothesis Significance Testing ◽  
P Values ◽  
Statistical Procedures ◽  
Major Section

There has been much debate about null hypothesis significance testing, p-values without null hypothesis significance testing, and confidence intervals. The first major section of the present article addresses some of the main reasons these procedures are problematic. The conclusion is that none of them are satisfactory. However, there is a new procedure, termed the a priori procedure (APP), that validly aids researchers in obtaining sample statistics that have acceptable probabilities of being close to their corresponding population parameters. The second major section provides a description and review of APP advances. Not only does the APP avoid the problems that plague other inferential statistical procedures, but it is easy to perform too. Although the APP can be performed in conjunction with other procedures, the present recommendation is that it be used alone.

The Numbers Will Love You Back in Return—I Promise

2016 ◽  
Vol 11 (4) ◽  
pp. 551-554 ◽  
Author(s):  
Martin Buchheit
Keyword(s):  
Sample Size ◽  
Null Hypothesis ◽  
Clinical Medicine ◽  
Statistical Significance ◽  
Significance Testing ◽  
Null Hypothesis Significance Testing ◽  
Sport Science ◽  
Size Dependent ◽  
Research Questions ◽  
Per Se

The first sport-science-oriented and comprehensive paper on magnitude-based inferences (MBI) was published 10 y ago in the first issue of this journal. While debate continues, MBI is today well established in sport science and in other fields, particularly clinical medicine, where practical/clinical significance often takes priority over statistical significance. In this commentary, some reasons why both academics and sport scientists should abandon null-hypothesis significance testing and embrace MBI are reviewed. Apparent limitations and future areas of research are also discussed. The following arguments are presented: P values and, in turn, study conclusions are sample-size dependent, irrespective of the size of the effect; significance does not inform on magnitude of effects, yet magnitude is what matters the most; MBI allows authors to be honest with their sample size and better acknowledge trivial effects; the examination of magnitudes per se helps provide better research questions; MBI can be applied to assess changes in individuals; MBI improves data visualization; and MBI is supported by spreadsheets freely available on the Internet. Finally, recommendations to define the smallest important effect and improve the presentation of standardized effects are presented.

Роль показника розміру ефекту в сучасних психологічних дослідженнях

2021 ◽  
Author(s):  
Валерій Боснюк
Keyword(s):  
Null Hypothesis ◽  
Significance Testing ◽  
P Value ◽  
Null Hypothesis Significance Testing

Для підтвердження результатів дослідження в психологічних наукових роботах протягом багатьох років використовується процедура перевірки значущості нульової гіпотези (загальноприйнята абревіатура NHST – Null Hypothesis Significance Testing) із застосуванням спеціальних статистичних критеріїв. При цьому здебільшого значення статистики «p» (p-value) розглядається як еквівалент важливості отриманих результатів і сили наукових доказів на користь практичного й теоретичного ефекту дослідження. Таке некоректне використання та інтерпретації p-value ставить під сумнів застосування статистики взагалі та загрожує розвитку психології як науки. Ототожнення статистичного висновку з науковим висновком, орієнтація виключно на новизну в наукових дослідженнях, ритуальна прихильність дослідників до рівня значущості 0,05, опора на статистичну категоричність «так/ні» під час прийняття рішення призводить до того, що психологія примножує тільки результати про наявність ефекту без врахування його величини, практичної цінності. Дана робота призначена для аналізу обмеженості p-value при інтерпретації результатів психологічних досліджень та переваг представлення інформації про розмір ефекту. Застосування розмірів ефекту дозволить здійснити перехід від дихотомічного мислення до оціночного, визначати цінність результатів незалежно від рівня статистичної значущості, приймати рішення більш раціонально та обґрунтовано. Обґрунтовується позиція, що автор наукової роботи при формулюванні висновків дослідження не повинен обмежуватися одним єдиним показником рівня статистичної значущості. Осмислені висновки повинні базуватися на розумному балансуванні p-value та інших не менш важливих параметрів, одним з яких виступає розмір ефекту. Ефект (відмінність, зв’язок, асоціація) може бути статистично значущим, а його практична (клінічна) цінність – незначною, тривіальною. «Статистично значущий» не означає «корисний», «важливий», «цінний», «значний». Тому звернення уваги психологів до питання аналізу виявленого розміру ефекту має стати обов’язковим при інтерпретації результатів дослідження.

scholarly journals Will the numbers really love you back: Re-examining Magnitude-based Inference

2017 ◽  
Author(s):  
Michael Lloyd Butson
Keyword(s):  
Sports Medicine ◽  
Null Hypothesis ◽  
Significance Testing ◽  
Null Hypothesis Significance Testing ◽  
Full Account ◽  
Sports Science ◽  
Theoretical Foundations

Many sports medicine and sports science researchers use Null Hypothesis Significance Testing despite it being criticized for being an amalgam of two irreconcilable methodologies. Hopkins and Batterham proposed Magnitude-based Inference as an alternative to Null Hypothesis Significance Testing; however, its validity and utility has also been questioned. Recently, it was suggested that the critics of Magnitude-based Inference lacked vision and that their objections to Magnitude-based inference should be ignored. However, a re-examination of Hopkins and Batterham’s explanation of their method indicates that they use profoundly different approaches in ways that are at odds with their theoretical foundations and intended purposes. If Hopkins and Batterham were to provide a full account of how their method is implemented, it could be comprehensively assessed. Until then, sports medicine and sports science researchers should use other theoretically valid methods that have had their utility established.

scholarly journals On the Potential Mismatch between the Function of the Bayes Factor and Researchers’ Expectations

2021 ◽  
Author(s):  
Tsz Keung Wong ◽  
Henk Kiers ◽  
Jorge Tendeiro
Keyword(s):  
Null Hypothesis ◽  
Bayes Factor ◽  
Survey Study ◽  
Significance Testing ◽  
Null Hypothesis Significance Testing ◽  
Posterior Odds ◽  
Statistical Tool ◽  
Reporting Practices ◽  
Insight Into

The aim of this study is to investigate whether there is a potential mismatch between the usability of a statistical tool and psychology researchers’ expectation of it. Bayesian statistics is often promoted as an ideal substitute for frequentists statistics since it coincides better with researchers’ expectations and needs. A particular incidence of this is the proposal of replacing Null Hypothesis Significance Testing (NHST) by Null Hypothesis Bayesian Testing (NHBT) using the Bayes factor. In this paper, it is studied to what extent the usability and expectations of NHBT match well. First, a study of the reporting practices in 73 psychological publications was carried out. It was found that eight Questionable Reporting and Interpreting Practices (QRIPs) occur more than once among the practitioners when doing NHBT. Specifically, our analysis provides insight into possible mismatches and their occurrence frequencies. A follow-up survey study has been conducted to assess such mismatches. The sample (N = 108) consisted of psychology researchers, experts in methodology (and/or statistics), and applied researchers in fields other than psychology. The data show that discrepancies exist among the participants. Interpreting the Bayes Factor as posterior odds and not acknowledging the notion of relative evidence in the Bayes Factor are arguably the most concerning ones. The results of the paper suggest that a shift of statistical paradigm cannot solve the problem of misinterpretation altogether if the users are not well acquainted with the tools.

Null Hypothesis Significance Testing: Ramifications, Ruminations and Recommendations

2005 ◽  
Vol 35 (1) ◽  
pp. 1-20 ◽  
Author(s):  
G. K. Huysamen
Keyword(s):  
Sample Size ◽  
Confidence Intervals ◽  
Effect Size ◽  
Null Hypothesis ◽  
Significance Testing ◽  
Population Parameter ◽  
Size Estimation ◽  
Null Hypothesis Significance Testing ◽  
Point Estimates ◽  
Size Estimates

Criticisms of traditional null hypothesis significance testing (NHST) became more pronounced during the 1960s and reached a climax during the past decade. Among others, NHST says nothing about the size of the population parameter of interest and its result is influenced by sample size. Estimation of confidence intervals around point estimates of the relevant parameters, model fitting and Bayesian statistics represent some major departures from conventional NHST. Testing non-nil null hypotheses, determining optimal sample size to uncover only substantively meaningful effect sizes and reporting effect-size estimates may be regarded as minor extensions of NHST. Although there seems to be growing support for the estimation of confidence intervals around point estimates of the relevant parameters, it is unlikely that NHST-based procedures will disappear in the near future. In the meantime, it is widely accepted that effect-size estimates should be reported as a mandatory adjunct to conventional NHST results.

Sign in / Sign up

Export Citation Format

Share Document