Investigating narrative performance in children with developmental language disorder: A systematic review and meta-analysis

Purpose: Speech-language pathologists (SLPs) typically examine narrative performance when completing a comprehensive language assessment. However, there is significant variability in the methodologies used to evaluate narration. The primary aims of this systematic review and meta-analysis were to a) investigate how narrative assessment type (e.g., macrostructure, microstructure, internal state language) differentiates typically developing (TD) children from children with developmental language disorder (DLD), or, TD–DLD group differences, b) identify specific narrative assessment measures (e.g., number of different words) that result in greater TD–DLD differences, and, c) evaluate participant and sample characteristics (e.g., DLD inclusionary criteria) that may uniquely influence performance differences. Method: Three electronic databases (PsychInfo, ERIC, and PubMed) and ASHAWire were searched on July 30, 2019 to locate studies that reported oral narrative language measures for both DLD and TD groups between ages 4 and 12 years; studies focusing on written narration or other developmental disorders only were excluded. Thirty-seven primary studies were identified via a three-step study selection procedure. We extracted data related to the sample participants, the narrative task(s) and assessment measures, and research design. Standardized mean differences using a bias-corrected Hedges’ g were the calculated effect sizes (N = 382). Research questions were analyzed using mixed-effects meta-regression with robust variance estimation to account for effect size dependencies. Results: Searches identified eligible studies published between 1987 and 2019. An overall meta-analysis using 382 effect sizes obtained across 37 studies showed that children with DLD had decreased narrative performance relative to TD peers, with summary estimates ranging from -0.850, 95% CI [-1.016, -0.685] to -0.794, 95% CI [-0.963, -0.624], depending on the correlation assumed. Across all models, effect size estimates showed significant heterogeneity both between and within studies, even after accounting for effect size-, sample-, and study-level predictors. Grammatical accuracy (microstructure) and story grammar (macrostructure) yielded the most consistent evidence of significant TD–DLD group differences across statistical models.Conclusions: Present findings suggest some narrative assessment measures may yield significantly different performance between children with and without DLD. However, researchers need to be consistent in their inclusionary criteria, their description of sample characteristics, and in their reporting of the correlations of measures, in order to determine which assessment measures are more likely to yield group differences.

Multi-source feedback following simulated resuscitation scenarios: a qualitative study

Background: The direct observation and assessment of learners’ resuscitation skills by an attending physician is challenging due to the unpredictable and time-sensitive nature of these events. Multisource feedback (MSF) may address this challenge and improve the quality of assessments provided to learners. We aimed to describe the similarities and differences in the assessment rationale of attending physicians, registered nurses, and resident peers in the context of a simulation-based resuscitation curriculum. Methods: We conducted a qualitative content analysis of narrative MSF of medical residents in their first postgraduate year of training who were participating in a simulation-based resuscitation course at two Canadian institutions. Assessments included an entrustment score and narrative comments from attending physicians, registered nurses, and resident peers in addition to self-assessment. Narrative comments were transcribed and analyzed thematically using a constant comparative method. Results: All 87 residents (100%) participating in the 2017-2018 course provided consent. A total of 223 assessments were included in our analysis. Four themes emerged from the narrative data: 1) Communication, 2) Leadership, 3) Demeanor, and 4) Medical Expert. Relative to other assessor groups, feedback from nurses focused on patient-centred care and communication while attending physicians focused on the medical expert theme. Peer feedback was the most positive. Self-assessments included comments within each of the four themes. Conclusions: In the context of a simulation-based resuscitation curriculum, MSF provided learners with different perspectives in their narrative assessment rationale and may offer a more holistic assessment of resuscitation skills within a competency-based medical education (CBME) program of assessment.

Early childhood care and education in the cultural context of Aotearoa (New Zealand)

© 2018 Taylor and Francis. Early childhood care and education in Aotearoa (New Zealand) has been celebrated through the international interest in the innovative sociocultural curriculum, Te Whāriki: He whāriki mātauranga mō ngā mokopuna o Aotearoa (New Zealand Ministry of Education, 1996). This document is now 20 years old, and is at the time of writing being updated by the New Zealand Ministry of Education. 1 In this chapter, a brief overview of the historical and cultural contexts of early childhood care and education leads into a discussion of some key cultural constructs and values that are recognised in Te Whāriki; in particular, those of the Indigenous people, the Māori. Discussion of the narrative assessment models that were developed to support the implementation of Te Whāriki is followed by an outline of implications for teacher education. The chapter ends with some reflections on aspirations for the future of early childhood care and education in Aotearoa.

Assessment in New Zealand Early Childhood Education: A Foucauldian Analysis

<p>This thesis aims to problematise and denaturalise the current dominant, empowerment infused early childhood education (ece) assessment discourse in Aoteaora New Zealand through a Foucauldian discourse analysis. It addresses a two-part question: How is contemporary ece assessment constructed in New Zealand, and, what is effected by this construction? Texts about contemporary ece assessment in New Zealand written by local ece scholars and practitioners as well as narrative assessment examples drawn from the Ministry of Education (2004) Kei Tua o te Pae, Assessment for Learning: Early Childhood Exemplars resource provide data for the analysis. The analysis is conducted in procedurally specified as well as open, associative, and playful modes. Contemporary ece assessment in New Zealand is found to be constructed as a new, post-developmental, morally desirable and secular salvation practice that is underpinned by principles of social justice, plurality and diversity. However, a consideration of key discursive truth-objects and their mobilisation within narrative assessments suggests that ece assessment may be implementing a boundless and normalising regime for the government of selves and others, and producing significant regulatory effects for children, teachers and whānau/ family. It is argued that ece assessment, as a technology of government, works to construct self responsible, self optimising, and permanently performing child-subjects. Such norms for self government map closely onto those that are promoted within neoliberal governmentalities. Ece assessment can therefore, at least in part, be understood as both a technique and effect of neoliberal rationalities of government. The ongoing status and dominant construction of ece assessment as an empowering, socially just practice is seen to be problematic. It stifles debate about early childhood spaces, and it is implicated in the constraint of multiple possibilities for the government of selves and others.</p>

Assessment in New Zealand Early Childhood Education: A Foucauldian Analysis

<p>This thesis aims to problematise and denaturalise the current dominant, empowerment infused early childhood education (ece) assessment discourse in Aoteaora New Zealand through a Foucauldian discourse analysis. It addresses a two-part question: How is contemporary ece assessment constructed in New Zealand, and, what is effected by this construction? Texts about contemporary ece assessment in New Zealand written by local ece scholars and practitioners as well as narrative assessment examples drawn from the Ministry of Education (2004) Kei Tua o te Pae, Assessment for Learning: Early Childhood Exemplars resource provide data for the analysis. The analysis is conducted in procedurally specified as well as open, associative, and playful modes. Contemporary ece assessment in New Zealand is found to be constructed as a new, post-developmental, morally desirable and secular salvation practice that is underpinned by principles of social justice, plurality and diversity. However, a consideration of key discursive truth-objects and their mobilisation within narrative assessments suggests that ece assessment may be implementing a boundless and normalising regime for the government of selves and others, and producing significant regulatory effects for children, teachers and whānau/ family. It is argued that ece assessment, as a technology of government, works to construct self responsible, self optimising, and permanently performing child-subjects. Such norms for self government map closely onto those that are promoted within neoliberal governmentalities. Ece assessment can therefore, at least in part, be understood as both a technique and effect of neoliberal rationalities of government. The ongoing status and dominant construction of ece assessment as an empowering, socially just practice is seen to be problematic. It stifles debate about early childhood spaces, and it is implicated in the constraint of multiple possibilities for the government of selves and others.</p>

Warnings in early narrative assessment that might predict performance in residency: signal from an internal medicine residency program

Introduction Narrative assessment data are valuable in understanding struggles in resident performance. However, it remains unknown which themes in narrative data that occur early in training may indicate a higher likelihood of struggles later in training, allowing programs to intervene sooner. Methods Using learning analytics, we identified 26 internal medicine residents in three cohorts that were below expected entrustment during training. We compiled all narrative data in the first 6 months of training for these residents as well as 13 typically performing residents for comparison. Narrative data were blinded for all 39 residents during initial phases of an inductive thematic analysis for initial coding. Results Many similarities were identified between the two cohorts. Codes that differed between typical and lower entrusted residents were grouped into two types of themes: three explicit/manifest and three implicit/latent with six total themes. The explicit/manifest themes focused on specific aspects of resident performance with assessors describing 1) Gaps in attention to detail, 2) Communication deficits with patients, and 3) Difficulty recognizing the “big picture” in patient care. Three implicit/latent themes, focused on how narrative data were written, were also identified: 1) Feedback described as a deficiency rather than an opportunity to improve, 2) Normative comparisons to identify a resident as being behind their peers, and 3) Warning of possible risk to patient care. Discussion Clinical competency committees (CCCs) usually rely on accumulated data and trends. Using the themes in this paper while reviewing narrative comments may help CCCs with earlier recognition and better allocation of resources to support residents’ development.

Developing a dashboard for faculty development in competency-based training programs: a design-based research project

Background: Canadian specialist residency training programs are implementing a form of competency-based medical education (CBME) that requires frequent assessments of entrustable professional activities (EPAs). Faculty struggle to provide helpful feedback and assign appropriate entrustment scores. CBME faculty development initiatives rarely incorporate teaching metrics. Dashboards could be used to visualize faculty assessment data to support faculty development. Methods: Using a design-based research process, we identified faculty development needs related to CBME assessments and designed a dashboard containing elements (data, analytics, and visualizations) meeting these needs. Data was collected within the emergency medicine residency program at the University of Saskatchewan through interviews with program leaders, faculty development experts, and faculty participating in development sessions. Two investigators thematically analyzed interview transcripts to identify faculty needs that were audited by a third investigator. The needs were described using representative quotes and the dashboard elements designed to address them. Results: Between July 1, 2019 and December 11, 2020 we conducted 15 interviews with nine participants (two program leaders, three faculty development experts, and four faculty members). Three needs emerged as themes from the analysis: analysis of assessments, contextualization of assessments, and accessible reporting. We addressed these needs by designing an accessible dashboard to present contextualized quantitative and narrative assessment data for each faculty member. Conclusions: We identified faculty development needs related to EPA assessments and designed dashboard elements to meet them. The resulting dashboard was used for faculty development sessions. This work will inform the development of CBME assessment dashboards for faculty.

The effect of workshop training on rater variability in children’s oral narrative assessment

Oral narrative assessments are important for diagnosis of language disorders in school-age children so scoring needs to be reliable and consistent. This study explored the impact of training on the variability of story grammar scores in children’s oral narrative assessments scored by multiple raters. Fifty-one speech pathologists and 19 final-year speech pathology students attended training workshops on oral narrative assessment scoring and analysis. Participants scored two oral narratives prompted by two different story stimuli and produced by two children of differing ages. Demographic information, story grammar scores and a confidence survey were collected pre- and post-training. The total story grammar score changed significantly for one of the two oral narratives. A significant effect was observed for rater years of experience and the change in total story grammar scores post training, with undergraduate students showing the greatest change. Two story grammar elements, character and attempt, changed significantly for both stories, with an overall trend of increased element scores post-training. Confidence ratings also increased post-training. Findings indicated that training via an interactive workshop can reduce rater variability when using researcher-developed narrative scoring systems.