Effective questionnaire design: How to use cognitive interviews to refine questionnaire items

BACKGROUND: Cognitive interviewing is a technique that can be used to improve and refine questionnaire items. We describe the basic methodology of cognitive interviewing and illustrate its utility through our experience using cognitive interviews to refine a questionnaire assessing parental understanding of concepts related to preterm birth. METHODS: Cognitive interviews were conducted using current best practices. Results were analyzed by the multidisciplinary research team and questionnaire items that were revealed to be problematic were revised. RESULTS: Revisions to the questionnaire items were made to improve clarity and to elicit responses that truly reflected the participants understanding of the concept. CONCLUSION: Cognitive interviewing is a useful methodology for improving validity of questionnaire items, we recommend researchers developing new questionnaire items design and complete cognitive interviews to improve their items and increase confidence in study conclusions.

Improving abortion underreporting in the United States: a cognitive interview study

Background: Abortion is a difficult-to-measure behavior with extensive survey underreporting, which compromises the ability to study and monitor abortion. The purpose of this study was to improve our understanding of how women interpret and respond to survey items asking if they have ever had an abortion. Methods: We developed multiple new questions hypothesized to improve abortion reporting, using approaches that aim to clarify which experiences to report as an induced abortion; reduce the stigma and sensitivity of induced abortion; reduce the sense of intrusiveness of asking about abortion; and increase the motivation to report. We conducted cognitive interviews with cisgender women aged 18-49 in two US states (N=64) to iteratively assess these new approaches and questions for improving abortion reporting. Results: Our findings suggest that including abortion as part of a list of other sexual and reproductive health services, asking a yes/no question about lifetime experience of abortion instead of asking about number of abortions, and developing an improved introduction to abortion questions may help to elicit more accurate survey reports. Conclusions: Opportunities exist to improve survey measurement of abortion. Reducing underreporting of abortion in surveys has the potential to improve sexual and reproductive health research that relies on pregnancy histories.

Item Pool Development for the School-Aged Life Impact Burn Recovery Evaluation (SA- LIBRE5-12) Profile Computerized Adaptive Test (CAT): An Observer-Reported Outcome Assessment Measuring the Impact of Burn Injuries in School-Aged Children

Introduction The transition from early childhood to teen years (5-12) is a critical time of development, which can be made particularly challenging by a burn injury. Assessing post-burn recovery during these years is important for improving pediatric survivors’ development and health outcomes. Few validated burn-specific measures exist for this age group. The purpose of this study was to generate item pools that will be used to create a future computerized adaptive test (CAT) assessing post-burn recovery in school-aged children. Methods Item pool development was guided by the previously developed School-Aged Life Impact Burn Recovery Evaluation (SA-LIBRE5-12) Conceptual Framework. The item pool development process involved a systematic literature review, extraction of candidate items from existing legacy measures, iterative item review during expert consensus meetings, and parent cognitive interviews. Results The iterative item review with experts consisted of six rounds. A total of 10 parent cognitive interviews were conducted. The three broad themes of concern were items that needed 1) clarification, needed context or were vague, 2) age dependence and relevance, and 3) word choice. The cognitive interviews indicated that survey instructions, recall period, item stem, and response choices were interpretable by respondents. Final item pool based on parental feedback consist of 57, 81, and 60 items in Physical, Psychological, and Family and Social Functioning respectively. Conclusion Developed item pools (n=198) in three domains are consistent with the existing conceptual framework. The next step involves field-testing the item pool and calibration using item response theory to develop and validate the SA-LIBRE5-12 CAT Profile.

Development and Validation of the RAFFLE; A Measure of Reasons and Facilitators for Loot Box Engagement

Qualitative studies have identified a diverse array of motivations for purchasing items within video games through chance-based mechanisms (i.e., “loot boxes”). Given that some individuals—particularly those at risk of disordered gaming and/or gambling—are prone to over-involvement with loot box purchasing, it is important to have a reliable, valid means of measuring the role of different motivations in driving purchasing behaviour. Building on prior qualitative research, this paper reports the development and validation of the “RAFFLE” scale, to measure the Reasons and Facilitators for Loot box Engagement. A 23-item, seven-factor scale was developed through cognitive interviews (n = 25) followed by two surveys of UK-based gamers who purchase loot boxes; analysed via exploratory (n = 503) and confirmatory (n = 1495) factor analysis, respectively. Subscales encompassed “enhancement’; “progression’; “social pressure’; “distraction/compulsion’; “altruism’; “fear of missing out’; and “resale”. The scale showed good criterion and construct validity (correlating well with measures of loot box engagement; the risky loot box index (r = 0.63) and monthly self-reported spend (r = 0.38)), and good internal validity (Cronbach’s alpha = 0.84). Parallels with, and divergence from, motivations for related activities of gaming and gambling, and alignment with broader theoretical models of motivation, are discussed.

Patients’ Experiences with Staphylococcus aureus and Gram-Negative Bacterial Bloodstream Infections: Results from Cognitive Interviews to Inform Assessment of Health-Related Quality of Life

Background We previously conducted a concept elicitation study on the impact of Staphylococcus aureus and Gram-negative bacterial bloodstream infections (SAB/GNB) on health-related quality of life (HRQoL) from the patient’s perspective and found significant impacts on HRQoL, particularly in physical and functional domains. Using this information and following guidance on the development of patient-reported outcome (PRO) measures, we determined which combination of measures and items (i.e., specific questions) would be most appropriate in a survey assessing HRQoL in bloodstream infections. Methods We selected a variety of measures/items from PROMIS ® (Patient-Reported Outcomes Measurement Information System ®) representing different domains. We purposefully sampled patients approximately 6-12 weeks post SAB/GNB and conducted two rounds of cognitive interviews to refine the survey by exploring patients’ understanding of items and answer selection as well as relevance for capturing HRQoL. Results We interviewed 17 SAB/GNB patients. Based on the first round of cognitive interviews (n=10), we revised the survey. After round two of cognitive interviewing (n = 7), we finalized the survey to include 10 different PROMIS ® short form measures of the most salient HRQoL domains and 2 adapted questions (41 items total), which were found to adequately capture HRQoL. Conclusion We developed a survey from well-established PRO measures that captures what matters most to SAB/GNB patients as they recover. This survey, uniquely tailored to bloodstream infections, can be used to assess these meaningful, important HRQoL outcomes in clinical trials and in patient care. Engaging patients is crucial to developing treatments for bloodstream infections.

Cognitive testing of a survey instrument to assess sexual practices, behaviours, and health outcomes: a multi-country study protocol

Background Population level data on sexual practices, behaviours and health-related outcomes can ensure that responsive, relevant health services are available for all people of all ages. However, while billions of dollars have been invested in attempting to improve sexual and reproductive health (including HIV) outcomes, far less is understood about associated sexual practices and behaviours. Therefore, the World Health Organization embarked on a global consultative process to develop a short survey instrument to assess sexual health practices, behaviours and health outcomes. In order for the resulting draft survey instrument to be published as a ‘global’ standard instrument, it is important to first determine that the proposed measures are globally comprehensible and applicable. This paper describes a multi-country study protocol to assess the interpretability and comparability of the survey instrument in a number of diverse countries. Methods This study will use cognitive interviewing, a qualitative data collection method that uses semi-structured interviews to explore how participants process and respond to survey instruments. We aim to include study sites in up to 20 countries. The study procedures consist of: (1) localizing the instrument using forward and back-translation; (2) using a series of cognitive interviews to understand how participants engage with each survey question; (3) revising the core instrument based on interview findings; and (4) conducting an optional second round of cognitive interviews. Data generated from interviews will be summarised into a predeveloped analysis matrix. The entire process (a ‘wave’ of data collection) will be completed simultaneously by 5+ countries, with a total of three waves. This stepwise approach facilitates iterative improvements and sharing across countries. Discussion An important output from this research will be a revised survey instrument, which when subsequently published, can contribute to improving the comparability across contexts of measures of sexual practices, behaviours and health-related outcomes. Site-specific results of the feasibility of conducting this research may help shift perceptions of who and what can be included in sexual health-related research.

Development of a scale to measure expected concussion reporting behavior

Background Most concussion education aims to increase athlete self-report of concussive symptoms. Although the population burden of concussion is high, frequency with which this injury occurs on a given sports team in a given season is relatively low. This means that powering concussion education evaluation studies to measure change in post-injury symptom reporting behavior requires what is often a prohibitively large sample size. Thus, evaluation studies are typically powered to measure proximal cognitions. Expected reporting behavior, a cognition that reflects planned and reactive decision-making, is a theoretically indicated construct for inclusion in evaluation studies. However, previously no scales were available to measure this construct with demonstrated reliability and validity among youth athletes. The objective of this study was to develop and assess the validity of a brief single-factor scale to measure expected youth athlete concussion reporting behavior (CR-E) in a sample of youth athletes. Methods A mixed methods approach was used, including cognitive interviews with youth athletes, and quantitative item reduction and validation. Participants were youth athletes (aged 9–16) from the Seattle metropolitan and rural south-Georgia regions. After refining an initial pool of items using cognitive interviews with a diverse group of youth athletes (n = 20), a survey containing these items was administered to youth soccer and football players (n = 291). Item reduction statistics and sequential confirmatory factor analyses were used to reduce the initial scale using a randomly selected half of the sample. Then, a final confirmatory factor analysis and validation tests were applied to the other half of the sample of youth athletes. Predictive validation was conducted longitudinally in a separate sample of youth athletes (n = 155). Results Internal consistency was high (alpha = 0.89), model fit was excellent, validation tests were in the hypothesized directions, and the scale was feasible to use. Using the finalized 4-item scale, we observed that less than one-third of youth soccer and football athletes expect to “always” tell their coach about symptoms of a suspected concussion. Conclusions The CR-E measure should be included in future studies evaluating concussion education programming in youth athlete populations.

Considerations for conducting and reporting digitally supported cognitive interviews with children and adults

Background Cognitive interviewing is a well-established qualitative method used to develop and refine PRO measures. A range of digital technologies including phone, web conferencing, and electronic survey platforms can be leveraged to support the conduct of cognitive interviewing in both children and adults. These technologies offer a potential solution to enrolling underrepresented populations, including those with rare conditions, functional limitations and geographic or socioeconomic barriers. In the aftermath of the COVID-19 pandemic, the use of digital technologies for qualitative interviewing will remain essential. However, there is limited guidance about adapting cognitive interviewing procedures to allow for remote data capture, especially with children. Methods Synthesizing the literature and our research experiences during the COVID-19 pandemic, we examine considerations for implementing digitally supported cognitive interviews with children, adolescents, and adults. We offer recommendations to optimize data quality and empirical rigor and illustrate the application of these recommendations in an ongoing cognitive interviewing study to develop and refine a new pediatric PRO measure. Results Good research practices must address participant and researcher preparation for study-related procedures and should anticipate and pre-emptively manage technological barriers. Field notes should detail interview context, audio/video cues, and any impact of technological difficulties on data quality. The approaches we recommend have been tested in an ongoing cognitive interviewing study that is enrolling children/adolescents with cGVHD ages 5–17 and their caregivers [NCT 04044365]. The combined use of telephone and videoconferencing to conduct cognitive interviews remotely is feasible and acceptable and yields meaningful data to improve the content validity of our new PRO measure of cGVHD symptom bother. Conclusion Digitally supported cognitive interviewing procedures will be increasingly employed. Remote data collection can accelerate accrual, particularly in multi-site studies, and may allow for interviewer personnel and data management to be centralized within a coordinating center, thus conserving resources. Research is needed to further test and refine techniques for remote cognitive interviewing, particularly in traditionally underrepresented populations, including children and non-English speakers. Expansion of international standards to address digitally supported remote qualitative data capture appears warranted.

Feasibility of Routine Quality-of-Life Assessment in Long-Term Care Homes

Maximizing long-term care (LTC) residents' quality of life (QoL) is the primary goal of care. However, most residents have cognitive impairment and care staff time is severely limited, leading to various complexities in measuring QoL. This study developed and assessed the feasibility of an approach to routinely measuring QoL in LTC residents. We used the DEMQOL-CH, a practical, reliable, valid tool, developed in the UK to be completed by care aides to assess QoL in residents with moderate to severe dementia. We recruited 45 care aides in 10 LTC homes in Alberta, Canada who we surveyed on the QoL of 263 residents via video calls. We assessed time to complete; care aide and manager perceived feasibility of completing the DEMQOL-CH; internal consistency and inter-rater reliability of DEMQOL-CH scores; and we conducted cognitive interviews with 7 care aides to assess care aide comprehension of the tool. Time to complete was on average 4 minutes with little variation. Care aides and managers rated using the DEMQOL-CH as highly feasible and valuable. The internal consistency of the DEMQOL-CH score was 0.80. The DEMQOL-CH score inter-rater agreement was 0.73. Cognitive interviews suggested good comprehension overall with some comprehension problems especially in care aides who speak English as a second language. Asking care aides to complete the DEMQOL-CH is highly feasible, requires minor resources, and reliability is high. However, some items caused comprehension and reliability problems. Reasons and possible solutions will be subject to further investigations.