Problems with evidence assessment in COVID-19 health policy impact evaluation: a systematic review of study design and evidence strength

IntroductionAssessing the impact of COVID-19 policy is critical for informing future policies. However, there are concerns about the overall strength of COVID-19 impact evaluation studies given the circumstances for evaluation and concerns about the publication environment.MethodsWe included studies that were primarily designed to estimate the quantitative impact of one or more implemented COVID-19 policies on direct SARS-CoV-2 and COVID-19 outcomes. After searching PubMed for peer-reviewed articles published on 26 November 2020 or earlier and screening, all studies were reviewed by three reviewers first independently and then to consensus. The review tool was based on previously developed and released review guidance for COVID-19 policy impact evaluation.ResultsAfter 102 articles were identified as potentially meeting inclusion criteria, we identified 36 published articles that evaluated the quantitative impact of COVID-19 policies on direct COVID-19 outcomes. Nine studies were set aside because the study design was considered inappropriate for COVID-19 policy impact evaluation (n=8 pre/post; n=1 cross-sectional), and 27 articles were given a full consensus assessment. 20/27 met criteria for graphical display of data, 5/27 for functional form, 19/27 for timing between policy implementation and impact, and only 3/27 for concurrent changes to the outcomes. Only 4/27 were rated as overall appropriate. Including the 9 studies set aside, reviewers found that only four of the 36 identified published and peer-reviewed health policy impact evaluation studies passed a set of key design checks for identifying the causal impact of policies on COVID-19 outcomes.DiscussionThe reviewed literature directly evaluating the impact of COVID-19 policies largely failed to meet key design criteria for inference of sufficient rigour to be actionable by policy-makers. More reliable evidence review is needed to both identify and produce policy-actionable evidence, alongside the recognition that actionable evidence is often unlikely to be feasible.

Mutual Learning for Policy Impact: Insights from CORE. Adapting research methods in the context of Covid-19

On 25 November 2021, the CORE Knowledge Translation Services team at the Institute of Development Studies, UK, hosted an online clinic session to facilitate the sharing of experiences and mutual learning on how CORE projects have or can adapt their research activities in the context of the Covid-19 pandemic. The clinic was attended by 22 CORE members from 12 projects and featured contributions from two CORE projects: The Youth Question in Africa: Impact, Response and Protection Measures in the IGAD Region and A New Digital Deal for an Inclusive Post-Covid-19 Social Compact: Developing Digital Strategies for Social and Economic Reconstruction. This learning guide captures the practical insights and advice from the event, to help inform the practice of participants and other projects across the portfolio.