Sampling Engagement in Shared Reading Activities With Children With Autism Spectrum Disorder: A Generalizability Study

Researchers seeking to make valid conclusions about engagement for young children with autism spectrum disorder (ASD) must first determine the reliability of estimates obtained across the conditions sampled. Working from that premise, we conducted a secondary data analysis of shared book readings between caregivers and their children with ASD, examining the contribution of measurement error on estimates of four states of child engagement. Caregivers read two different book types, each three times, with their children. With book type and occasion as measurement facets, we conducted a generalizability study and a series of decision studies. With the interaction of Persons × Book Type × Occasion contributing most measurement error for four engagement variables, we only found stable estimates for unengaged behaviors. For the variables of active engagement, visual engagement, and disruptive, four, five, and more than 10 book types, respectively, were necessary to obtain stable estimates across two occasions.

Assessing the validity of an OSCE developed to assess rare, emergent or complex clinical conditions in endocrinology & metabolism

Background Assessment of emergent, rare or complex medical conditions in Endocrinology and Metabolism (E&M) is an integral component of training. However, data is lacking on how this could be best achieved. The purpose of this study was to develop and administer an Objective Structured Clinical Examination (OSCE) for E&M residents, and to gather validity evidence for its use. Methods A needs assessment survey was distributed to all Canadian E&M Program Directors and recent graduates to determine which topics to include in the OSCE. The top 5 topics were selected using a modified Delphi technique. OSCE cases based on these topics were subsequently developed. Five E&M residents (PGY4-5) and five junior Internal Medicine (IM) residents participated in the OSCE. Performance of E&M and IM residents was compared and results were analyzed using a Generalizability study. Examiners and candidates completed a survey following the OSCE to evaluate their experiences. Results The mean score of IM and E&M residents was 41.7 and 69.3 % (p < 0.001), respectively, with a large effect size (partial η2 = 0.75). Overall reliability of the OSCE was 0.74. Standard setting using a borderline regression method resulted in a pass rate of 100 % of E&M residents and 0 % of IM residents. All residents felt the OSCE had high value for learning as a formative exam. Conclusions The E&M OSCE is a feasible method for assessing emergent, rare and complex medical conditions and this study provides validity evidence to support its use in a competency-based curriculum.

A Preliminary Characterization of Canonicalized and Non-Canonicalized Section Headers Across Variable Clinical Note Types

In the electronic health record, the majority of clinically relevant information is stored within clinical notes. Most clinical notes follow a set organizational structure composed of canonicalized section headers that facilitate clinical review and information gathering. Standardized section header terminologies such as the SecTag terminology permit the identification and standardization of headers to a canonicalized form. Although the SecTag terminology has been evaluated extensively for history & physical notes, the coverage of canonical section header terms has not been assessed across other note types. For this pilot study, we conducted a coverage study and characterization of canonical section headers across 5 common, clinical note types and a generalizability study of canonical section headers detected within two types of clinical notes from Penn Medicine.

Validity, reliability and feasibility of assessment of cilinical reasoning of medical students by observation versus post encounter assessment in a clinical practice setting.

Background The assessment of clinical reasoning by medical students in clinical practice is very difficult. Partly this is because the fundamental mechanisms of clinical reasoning are difficult to uncover and when known, hard to observe and interpret. Methods We developed an observation tool to assess the clinical reasoning ability of medical students during clinical practice. The observation tool consists of an 11-item observation rating form. The validity, reliability and feasibility of this tool were verified among medical students during the internal medicine clerkship and compared to an existing post-encounter rating tool. Results Six raters assessed each, the same 15 student-patient encounters. The internal consistency (Cronbach’s alfa) for the observation rating tool (ORT) was 0.87 (0.71-0.84) and the 5-item post encounter rating tool (PERT) was 0.81 (0.71-0.87). The intraclass-correlation coefficient for single measurements was poor for both the ORT; 0.32 (p<0.001) as well as the PERT; 0.36 (p<0.001). The generalizability study (G-study) and decision study (D-study) showed that 6 raters are required to achieve a G-coefficient of > 0.7 for the ORT and 7 raters for the PERT. The largest sources of variance are the interaction between raters and students. There was a correlation between the ORT and PERT of 0.53 (p=0.04) Conclusions The ORT and PERT are both feasible, valid and reliable instruments to assess students’ clinical reasoning skills in clinical practice.

Sistema de observación para analizar la interacción en el juego de Boccia por equipos

Se ha diseñado un instrumento de observación ad hoc, combinación de formato de campo y sistemas de categorías, que permite analizar la interacción social -conductas prosociales y antisociales- que tiene lugar en la competición por equipos en el juego de boccia. El registro y codificación de los datos se ha desarrollado mediante el software Lince. La validez de contenido del instrumento de observación ha quedado avalada por el equipo técnico de la Selección Española de boccia. Los resultados relativos a la concordancia entre los registros generados por tres observadores diferentes, calculada mediante el coeficiente Kappa de Cohen, indican una elevada fiabilidad de los datos obtenidos mediante el sistema de observación. En el seno de la teoría de la Generalizabilidad, mediante el software SAGT, se ha desarrollado el plan de medida [Jugador] [Categoría] / [Parcial], que ha permitido asegurar que, con el número de parciales analizados, se consigue una elevada fiabilidad de precisión de generalización. Además, se ha procedido a la optimización del plan de medida [Parciales] [Categorías] / [Jugador]. La operatividad del sistema de observación desarrollado ha quedado patente en los T-patterns detectados mediante el software Theme, versión 6. Edu. De los resultados obtenidos se desprende que el juego de boccia constituye un entorno favorable de elevado valor formativo para el colectivo de la discapacidad. The observation instrument was purpose-built and combines a field format with systems of categories. The observation instrument allows to analyze the social interaction -prosocial and antisocial behaviors- that takes place in team boccia competition. The content validity of the observation instrument has been guaranteed by the coaching staff of the Boccia Spanish Team. The data were coded with the Lince software programme. Cohen's Kappa coefficient obtained by comparing the data sets generated by three observers indicates a high reliability of the data. We also performed a generalizability study, [Player][Category]/[End], demonstrating the consistency of the data based on the Ends observed. The application of the optimization module for [End][Category]/[Player] facets showed us how many players would constitute an optimal sample in future studies. The practical application of the observation system was demonstrated by performing T-pattern analysis using Theme software programme. The results obtained show that boccia is a very favorable educational environment for the disability group. O instrumento de observação foi construído ad hoc e combina um formato de campo com sistemas de categorias. O instrumento de observação permite analisar a interação social - comportamentos anti-sociais e anti-sociais - que ocorre na competição de bocha em equipe. A validade de conteúdo do instrumento de observação foi garantida pela equipe técnica da Equipe Espanhola de Boccia. Os dados foram codificados com o programa de software Lince. Coeficiente Kappa de Cohen obtido pela comparação dos conjuntos de dados gerados por três observadores indica alta confiabilidade dos dados. Também realizamos um estudo de generalização [Jogador] [Categoria] / [Parcial], demonstrando a consistência dos dados com base nas extremidades observadas. A aplicação do módulo de otimização para as facetas [Parciales] [Categorias] / [Jogador] nos mostrou quantos jogadores seriam uma ótima amostra em estudos futuros. A aplicação prática do sistema de observação foi demonstrada através da análise do padrão T usando o programa de software Theme. Os resultados obtidos são desprezíveis que o jogo de bocha é constituído por um formulário de valor favorável para o colectivo da discapacidade.

Assessing the Reliability of the CAMS Rating Scale Using a Generalizability Study

Background: An important consideration when conducting randomized controlled trials is treatment differentiation. Direct observation helps ensure that providers in different treatment groups are delivering distinct interventions. One direct observation method is the use of a measure to rate clinician performance when delivering an intervention. Aims: This generalizability study evaluated the reliability of the CAMS Rating Scale (CRS), a measure used to assess delivery of the Collaborative Assessment and Management of Suicidality (CAMS). Method: Digitally recorded tapes of clinicians delivering either CAMS or Enhanced Care-As-Usual (E-CAU) were coded using the CRS. Sessions ( N = 36) were each coded by two raters, and encompassed four clinicians, four time points, and 34 unique patients across two treatment groups. A reliability coefficient (i.e., G coefficient) and the percentages of variance contributed by each component of the measurement model were obtained. Results: The CRS reliably differentiates CAMS from E-CAU, minimizes measurement error relative to expected variance sources, and continues to demonstrate high inter-rater reliability. Limitations: The absence of blind raters, a formal training protocol for the rating team, and ratings from all clinician–patient dyads at all time points was a limitation. Conclusion: The CRS is a reliable treatment differentiation measure that can play an integral role in studies evaluating CAMS.

Screening for technical flaws in multiple-choice items. A generalizability study.

Construction errors in multiple-choice items are quite prevalent and constitute threats to test validity of multiple-choice tests. Currently very little research on the usefulness of systematic item screening by local review committees before test administration seem to exist. The aim of this study was therefore to examine validity and feasibility aspects of review committee screening for item flaws. We examined the reliability of item reviewers’ independent judgments of the presence/absence of item flaws with a generalizability study design and found only moderate reliability using five reviewers. Statistical analyses of actual exam scores could be a more efficient way of identifying flaws and improving average item discrimination of tests in local contexts. The question of validity of human judgments of item flaws is important - not just for sufficiently sound quality assurance procedures of tests in local test contexts - but also for the global research on item flaws.

Haitian Children’s Disaster Trauma: Validation of Pictorial Assessment of Resilience and Vulnerability

The House-Tree-Person (HTP) drawing test has been culturally adapted for Haitian children and objectively scored for resilience and vulnerability (Roysircar, Colvin, Afolayan, Thompson, & Robertson, 2017). The HTP was used to assess 88 Haitian children’s adjustment to the 2010 earthquake and the continuous trauma of societal inequalities. The study examined the validity of the adapted HTP test and its dimensionality. The study included participant interviews with child self-report measures of self-esteem, as perceived by self, peers, and family; posttraumatic symptoms; and self-concept. All measures were translated and administered in Créole. Analyses included standardized sample scores; descriptive statistics; internal consistency reliability; interscale correlations; a generalizability study showing that there were no differences in HTP scores due to novice or expert raters; and an exploratory factor analysis of HTP scores indicating three factors and accounting for just under 50% of the variance. The three dimensions, HTP Resilience-Vulnerability Integrated, House Feeling Safe, and Person Feeling Unloved, are discussed within the international literature on child disaster trauma assessed pictorially, and within Haitians’ spiritual worldview of suffering and endurance.

A modification of the Schutte Emotional Intelligence Scale using multivariate generalizability theory

We modified the Schutte Emotional Intelligence Scale (EIS) using multivariate generalizability theory (MGT), a measurement system in which the focus is on multiple universe scores within a specific generalization universe. Application of MGT comprises conducting a generalizability study to analyze and estimate the associations among various factors, the number of factors, and the initial measurement design process, as well as a decision study to explore changes in the generalized coefficient and reliability indices when the number of items in the measure is changed. We conducted a generalizability study with 210 students from Shanghai and Anhui Province China, and performed item analysis and exploratory factor analysis, then modified the scale using MGT to create a modified edition of the EIS and tested item and factor reliability and validity with another 1,339 students from Shanghai and Anhui Province. Results revealed better reliability and validity than the original EIS, with each factor making a more balanced contribution to the total variance. The modified Chinese EIS will be useful for future emotional intelligence research.