Measuring implementation fidelity in a cluster-randomized pragmatic trial: development and use of a quantitative multi-component approach

Background In pragmatic trials, on-site partners, rather than researchers, lead intervention delivery, which may result in implementation variation. There is a need to quantitatively measure this variation. Applying the Framework for Implementation Fidelity (FIF), we develop an approach for measuring variability in site-level implementation fidelity. This approach is then applied to measure site-level fidelity in a cluster-randomized pragmatic trial of Music & MemorySM (M&M), a personalized music intervention targeting agitated behaviors in residents living with dementia, in US nursing homes (NHs). Methods Intervention NHs (N = 27) implemented M&M using a standardized manual, utilizing provided staff trainings and iPods for participating residents. Quantitative implementation data, including iPod metadata (i.e., song title, duration, number of plays), were collected during baseline, 4-month, and 8-month site visits. Three researchers developed four FIF adherence dimension scores. For Details of Content, we independently reviewed the implementation manual and reached consensus on six core M&M components. Coverage was the total number of residents exposed to the music at each NH. Frequency was the percent of participating residents in each NH exposed to M&M at least weekly. Duration was the median minutes of music received per resident day exposed. Data elements were scaled and summed to generate dimension-level NH scores, which were then summed to create a Composite adherence score. NHs were grouped by tercile (low-, medium-, high-fidelity). Results The 27 NHs differed in size, resident composition, and publicly reported quality rating. The Composite score demonstrated significant variation across NHs, ranging from 4.0 to 12.0 [8.0, standard deviation (SD) 2.1]. Scaled dimension scores were significantly correlated with the Composite score. However, dimension scores were not highly correlated with each other; for example, the correlation of the Details of Content score with Coverage was τb = 0.11 (p = 0.59) and with Duration was τb = − 0.05 (p = 0.78). The Composite score correlated with CMS quality star rating and presence of an Alzheimer’s unit, suggesting face validity. Conclusions Guided by the FIF, we developed and used an approach to quantitatively measure overall site-level fidelity in a multi-site pragmatic trial. Future pragmatic trials, particularly in the long-term care environment, may benefit from this approach. Trial registration Clinicaltrials.gov NCT03821844. Registered on 30 January 2019, https://clinicaltrials.gov/ct2/show/NCT03821844.

Effectiveness of a parent-mediated intervention for toddlers with autism spectrum disorder: Evidence from a large community implementation

Recent efforts have focused on developing and evaluating early intervention for toddlers with probable or emerging autism spectrum disorder. Parent-mediated approaches have gained traction, with mounting evidence of efficacy, but a research-to-practice gap exists, and community effectiveness remains to be firmly established. We report outcomes of a parent-mediated toddler intervention delivered through a research-community partnership, using a community-partnered participatory framework. Data were available for 179 of 183 toddler-parent dyads receiving Social ABCs parent coaching (mean toddler age: 25.18 months; range, 14–34 months). Of these, 89.4% completed the 12-week program and 70.6% returned for 3-month follow-up assessment. Parents attained implementation fidelity exceeding 75%, and toddlers made gains on proximal and distal measures of social communication. Parent fidelity was associated with toddlers’ responsivity at week 12, and responsivity predicted later language gains and reduced autism spectrum disorder symptoms. The roles of child, family, and system factors are discussed. Community delivery of an evidence-based parent-mediated intervention for toddlers with autism spectrum disorder is feasible and effective. Given resource efficiencies associated with parent-mediated approaches, findings bolster current efforts to promote earlier and more widespread access to intervention at the first signs of developmental concern. Lay abstract In an effort to increase access to intervention as early as possible for toddlers with autism spectrum disorder or signs thereof, many researchers have developed interventions that can be delivered by parents in their own homes. These parent-mediated approaches have gained a lot of research attention in recent years and have been found to be helpful in terms of parent and toddler learning. Several studies have used a rigorous research design (a randomized controlled trial) to show that parent-mediated intervention can work under ideal well-controlled conditions. To build on this evidence, we also need to examine whether parent-mediated interventions can be taught well through community service providers and delivered in more “real-world” conditions. This study used a research-community partnership to provide a parent-mediated intervention (called the Social ABCs) to 179 families (mean toddler age was 25 months; ranging from 14 to 34 months). Almost 90% of the families completed the 12-week program and 70% returned for a follow-up assessment 3 months later. Analyses showed that parents learned the strategies that were designed to help them support their toddlers’ development. Also, toddlers made gains in their language, communication, and social skills. Importantly, parents’ use of the strategies was related to toddlers’ skill gains, suggesting that the use of the strategies made a difference for the toddlers. Findings support the use of parent-mediated intervention in this very young age group and suggest that such intervention approaches should be made available for community delivery.

Implementation evaluation of a collective impact initiative to promote adolescent health in Oklahoma County, USA

Background The teenage birth rate in the USA has considerably decreased in recent decades; however, more innovative, collaborative approaches are needed to promote adolescent health and prevent teenage pregnancy at the community level. Despite literature on the promising results of the collective impact (CI) model for health promotion, there is limited literature on the model’s ability to reduce teenage pregnancies in a community. The Central Oklahoma Teen Pregnancy Prevention Collaboration is applying the CI model to foster collaboration among multiple stakeholders with the goal of increasing community and organizational capacity to improve adolescent health outcomes. This paper reports the findings from the initiative’s implementation evaluation, which sought to understand whether the CI model improved collaboration among organizations and understand barriers and facilitators that affected program delivery. Methods Program implementers and evaluators jointly developed research questions to guide the intervention and evaluation design. The Consolidated Framework for Implementation Research (CFIR) was used to assess program components including the intervention characteristics, organization setting, community setting, facilitator characteristics, and the process of implementation. Primary sources of data included performance measures, meeting observations (n = 11), and semi-structured interviews (n = 10). The data was thematically analyzed using CFIR constructs, community capacity domains, and the five constructs of CI. Results Key findings include the need for shortened meeting times for meaningful engagement, opportunities for organizations to take on more active roles in the Collaboration, and enhanced community context expertise (i.e., those with lived experience) in all Collaboration initiatives. We identified additional elements to the core constructs of CI that are necessary for successful implementation: distinct role identification for partner organizations and incorporation of equity and inclusivity into collaboration processes and procedures. Conclusions Results from this implementation evaluation provide valuable insights into implementation fidelity, participant experience, and implementation reach of an innovative, systems-level program. Findings demonstrate the context and requirements needed to successfully implement this innovative program approach and CI overall. Additional core elements for CI are identified and contribute to the growing body of literature on successful CI initiatives.

Sustained positive behaviour change of wounded, injured and sick UK military following an adaptive adventure sports and health coaching recovery course

IntroductionA rising trend has occurred in the physical and mental health challenges faced by recovering UK service personnel. To support these individuals, bespoke inclusive multiactivity and adventurous training courses (MAC) have been developed. This study investigated the MAC’s influence on participants’ ability to sustain day-to-day changes that facilitate positive mental health and psychological need satisfaction.MethodsThe 146 UK service personnel who participated in this study attended a five-day MAC 12 months ago. To investigate how the supportive experience influenced participants’ lives, quantitative and qualitative data were collected via an online survey. Open-ended questioning and abductive analysis were conducted to understand mechanisms, influential aspects of the course and positive behaviour change.ResultsPositive behaviour changes were reported by 74% of the respondents. These changes align with positive psychological well-being (98%). Impactful elements of the course experienced by participants mostly aligned with the three basic psychological needs of autonomy (34%), competence (36%) and relatedness (61%).ConclusionsRecovery support programmes that encompass health coaching adventurous activities, such as the MAC, can initiate long-term positive behaviour change for recovering military personnel. In this specific context, the concurrence of the self-determination theory concepts that underpin the course delivery and participant outcomes is a powerful endorsement of implementation fidelity.

Evaluation of the impact and implementation of a national clinical programme for the management of self-harm in hospital emergency departments: study protocol for a natural experiment

BackgroundA National Clinical Programme for the Management of Hospital-Presenting Self-Harm (NCP-SH) was introduced in Ireland in 2014. This involved the development of a model of care to standardise the management of self-harm in emergency departments, to be delivered by dedicated clinical nurse specialists. The core components of the programme were to: ensure an empathic and timely response, conduct a biopsychosocial assessment, involve family members in assessment and discharge planning, and provide a bridge to next care. The overall aim of the programme was to reduce the rate of repeat self-harm. This multistage study will evaluate the impact of the NCP-SH on hospital-presenting self-harm and to identify determinants influencing its implementation.MethodsEmploying a sequential mixed methods design, the first stage will use data from the National Self-Harm Registry Ireland to examine the impact of the NCP-SH on self-harm repetition, along with other aspects of care, including provision of psychosocial assessments and changes in admissions and postdischarge referrals. A cost-effectiveness analysis will assess the cost per repeat self-harm attendance avoided as a result of the NCP-SH. The second stage will identify the influences of implementation fidelity—adherence to the programme’s core components—using a combination of document analysis and semistructured interviews with staff of the programme, guided by the Consolidated Framework for Implementation Research.Ethics and disseminationThis study has received full ethical approval and will run until August 2023. This study is novel in that it will identify important factors influencing successful implementation of complex programmes. It is expected that the findings will provide important learnings for the integration of mental health services in general hospital settings and will be disseminated via peer-review publications along with reports for clinicians and policy-makers.

Can SBAR be implemented with high fidelity and does it improve communication between healthcare workers? A systematic review

ObjectiveTo characterise the extent to which health professionals perform SBAR (situation, background, assessment, recommendation) as intended (ie, with high fidelity) and the extent to which its use improves communication clarity or other quality measures.Data sourcesMedline, Healthstar, PsycINFO, Embase and CINAHL to October 2020 and handsearching selected journals.Study selection and outcome measuresEligible studies consisted of controlled trials and time series, including simple before-after design, assessing SBAR implementation fidelity or the effects of SBAR on communication clarity or other quality measures (eg, safety climate, patient outcomes).Data extraction and synthesisTwo reviewers independently abstracted data according to Preferred Reporting Items for Systematic Reviews and Meta-Analyses on study features, intervention details and study outcomes. We characterised the magnitude of improvement in outcomes as small (<20% relative increase), moderate (20%–40%) or large (>40%).ResultsTwenty-eight studies (3 randomised controlled trials, 6 controlled before-after studies, and 19 uncontrolled before-after studies) met inclusion criteria. Of the nine studies assessing fidelity of SBAR use, four occurred in classroom settings and three of these studies reported large improvements. The five studies assessing fidelity in clinical settings reported small to moderate effects. Among eight studies measuring communication clarity, only three reported large improvements and two of these occurred in classroom settings. Among the 17 studies reporting impacts on quality measures beyond communication, over half reported moderate to large improvements. These improvements tended to involve measures of teamwork and culture. Improvements in patient outcomes occurred only with intensive multifaceted interventions (eg, early warning scores and rapid response systems).ConclusionsHigh fidelity uptake of SBAR and improvements in communication clarity occurred predominantly in classroom studies. Studies in clinical settings achieving impacts beyond communication typically involved broader, multifaceted interventions. Future efforts to improve communication using SBAR should first confirm high fidelity uptake in clinical settings rather than assuming this has occurred.PROSPERO registration numberCRD42018111377.

Intergenerational Tutoring: Older Adults Supporting Children’s Academic Needs via Video Conferencing

COVID-19 resulted in societal disruptions across the lifespan. School (K-12) closures were among the most challenging impacts of the virus, leaving many parents with the burden of schooling their children at home. Another major impact of the virus was the social isolation and loneliness felt by many retired, older adults, who were sheltering at home. The disruptions of COVID-19 led our inter-professional team to develop the Intergenerational Tutoring program. Intergenerational Tutoring addresses a service delivery gap in schools because tutors expand schools' capacity to implement evidence-based instruction with students in need of individual support. At the same time, research shows that meaningful volunteering supports the well-being of older adults across physical, psychosocial and cognitive dimensions of health. The aim of the Intergenerational Tutoring program is to pair older adults with kindergarten children in high needs schools to implement early literacy interventions remotely via Zoom. Our poster will describe the Intergenerational Tutoring program including tutor training and tutoring implementation. We will summarize the initial findings from our pilot study conducted in spring and summer 2021 with tutors and children. Data will include (1) themes from tutor interviews regarding the personal meaningfulness of the program and the program’s associated benefits and challenges; (2) implementation fidelity data; (3) impact of tutoring on children’s early literacy skills; and (4) parent feedback. We will summarize lessons learned and next steps for the program.

A peek behind the curtain: exploring coaching styles within the implementation and sustainment facilitation (ISF) strategy in the substance abuse treatment to HIV care study

Background The Grasha-Riechmann teaching styles, which includes three didactic and two prescriptive styles, have been shown to help enhance learning within educational settings. Although an adaption of the Grasha-Riechmann style classification has enabled coaching styles to be identified for use as part of quality improvement (QI) initiatives, research has not examined the styles actually utilized by coaches within a QI initiative or how the styles change overtime when the coach is guiding an organization through change implementation. Interactions between coaches and HIV service organization (HSO) staff participating in a large implementation research experiment called the Substance Abuse Treatment to HIV care (SAT2HIV) Project were evaluated to begin building an evidence base to address this gap in implementation research. Methods Implementation & Sustainment Facilitation (ISF) Strategy meetings (n = 137) between coaches and HSO staff were recorded and professionally transcribed. Thematic coding classifications were developed from the Grasha-Riechmann framework and applied to a purposively selected sample of transcripts (n = 66). Four coders independently coded transcripts using NVivo to facilitate text identification, organization, and retrieval for analysis. Coaching style use and changes across the three ISF phases were explored. Results Facilitator and formal authority were the two coaching styles predominately used. Facilitator sub-themes shifted from asking questions and providing support to supporting independent action over time. Coaches’ use of formal authority sub-styles shifted notably across time from setting expectations or ensuring preparation to offering affirmation or feedback about changes that the HSO’s were implementing. The use of the delegator or personal model coaching styles occurred infrequently. Conclusions The current research extends implementation research’s understanding of coaching. More specifically, findings indicate it is feasible to use the Grasha-Riechmann framework to qualitatively identify coaching styles utilized in a facilitation-based implementation strategy. More importantly, results provide insights into how different coaching styles were utilized to implement an evidence-based practice. Further research is needed to examine how coaching styles differ by organization, impact implementation fidelity, and influence both implementation outcomes and client outcomes. Trial registration ClinicalTrials.gov NCT02495402. Registered on July 6, 2015.

Key Factors in Intervention Implementation, Fidelity, and Sustainability

As part of an NIA-supported effort to develop an online course to train individuals to lead the evidence-based Savvy Caregiver program and to orient sponsoring organizations to the program, we conducted semi-structured interviews to assess success and sustainability “best practices”. Interviews were conducted with 17 leaders and trainers from eleven Savvy-providing organizations. Analysis of these interviews identified two main themes associated with successful program implementation: leadership commitment and trainer ownership. Paramount to success appears to be leaders’ clear understanding of and enthusiastic commitment to the value of the Savvy program to the organization’s constituents. This translated to careful selection, training, management and on-going development of Savvy program trainers. It contributed to leaders’ appreciation of Savvy as a gateway for clients to seek out other programs and services from the organization, while the gathering of meaningful evaluation data (using established outcome-assessment instruments) contributed, in several cases, to garnering more lasting support to deliver the program. Organizations’ commitment to the program was demonstrated by securing the kinds of adequate and appropriate training, typically involving both instruction and modeling, for Savvy program leaders. These efforts fostered a sense of ownership among the leaders – the sense that the program had positive value for the caregivers served. These findings should be of interest both to scholars engaged in the development of interventions and for organizations implementing them. Taken together, the themes highlight several factors for program implementation that maximize the chances of maintaining fidelity to core program principles and ensuring its sustainability.