scholarly journals Reliability of Physiological Responses Induced by Basic Emotions: A Pilot Study

2019 ◽  
Vol 38 (1) ◽  
Author(s):  
Eun-Hye Jang ◽  
Sangwon Byun ◽  
Mi-Sook Park ◽  
Jin-Hun Sohn
Keyword(s):  
Internal Consistency ◽  
Interrater Reliability ◽  
Physiological Responses ◽  
Skin Conductance Level ◽  
Intraclass Correlation ◽  
Repeated Measurements ◽  
Physiological Signals ◽  
Basic Emotions ◽  
Reliability Indices ◽  
Film Clips

Abstract Background Although emotion-specific autonomic responses based on the discrete theory of emotion have been widely studied, studies on the reliability of physiological responses to emotional stimuli are limited. In this study, we aimed to assess the reliability of physiological changes induced by the six basic emotions (happiness, sadness, anger, fear, disgust, and surprise) that were measured during 10 weekly repeated experiments. Methods Twelve college students participated, and in each experiment, physiological signals were collected before and while participants were watching emotion-provoking film clips. Additionally, the participants self-evaluated the emotions that they experienced during the film presentation at the end of each emotional stimulus. To avoid adaptation of participants to identical stimuli during repeated measurements, we used 10 different film clips for each emotion, and thus a total of 60 film clips over 10 weeks were used. Physiological features, such as skin conductance level (SCL), fingertip temperature (FT), heart rate (HR), and blood volume pulse (BVP), were extracted from the physiological signals. Two reliability indices, Cronbach’s alpha and intraclass correlation coefficient, were calculated from the physiological features to assess internal consistency and interrater reliability, respectively. Results We found that SCL, HR, and BVP measured during the emotion-provoking phase over the 10 weekly sessions were more reliable than those assessed at baseline. Furthermore, SCL, HR, and BVP from the emotion-provoking phase exhibited excellent internal consistency and interrater reliability. Conclusions Our findings suggest that these features can be used as reliable physiological indices in emotion studies. The results also support the significance of physiological signals as meaningful indicators for emotion recognition in HCI (human computer interface) area.

scholarly journals Validation of an Evidence-Based Medicine Critically Appraised Topic Presentation Evaluation Tool (EBM C-PET)

2013 ◽  
Vol 5 (2) ◽  
pp. 252-256 ◽  
Author(s):  
Hans B. Kersten ◽  
John G. Frohna ◽  
Erin L. Giudice
Keyword(s):  
Internal Consistency ◽  
Evidence Based Medicine ◽  
Interrater Reliability ◽  
Intraclass Correlation ◽  
Correlation Coefficients ◽  
Evidence Based ◽  
Evaluation Tool ◽  
Intraclass Correlation Coefficients ◽  
Excellent Internal Consistency ◽  
Based Medicine

Abstract Background Competence in evidence-based medicine (EBM) is an important clinical skill. Pediatrics residents are expected to acquire competence in EBM during their education, yet few validated tools exist to assess residents' EBM skills. Objective We sought to develop a reliable tool to evaluate residents' EBM skills in the critical appraisal of a research article, the development of a written EBM critically appraised topic (CAT) synopsis, and a presentation of the findings to colleagues. Methods Instrument development used a modified Delphi technique. We defined the skills to be assessed while reviewing (1) a written CAT synopsis and (2) a resident's EBM presentation. We defined skill levels for each item using the Dreyfus and Dreyfus model of skill development and created behavioral anchors using a frame-of-reference training technique to describe performance for each skill level. We evaluated the assessment instrument's psychometric properties, including internal consistency and interrater reliability. Results The EBM Critically Appraised Topic Presentation Evaluation Tool (EBM C-PET) is composed of 14 items that assess residents' EBM and global presentation skills. Resident presentations (N  =  27) and the corresponding written CAT synopses were evaluated using the EBM C-PET. The EBM C-PET had excellent internal consistency (Cronbach α  =  0.94). Intraclass correlation coefficients were used to assess interrater reliability. Intraclass correlation coefficients for individual items ranged from 0.31 to 0.74; the average intraclass correlation coefficients for the 14 items was 0.67. Conclusions We identified essential components of an assessment tool for an EBM CAT synopsis and presentation with excellent internal consistency and a good level of interrater reliability across 3 different institutions. The EBM C-PET is a reliable tool to document resident competence in higher-level EBM skills.

scholarly journals Assessing Balance Function in Patients With Total Knee Arthroplasty

2015 ◽  
Vol 95 (10) ◽  
pp. 1397-1407 ◽  
Author(s):  
Andy C.M. Chan ◽  
Marco Y.C. Pang
Keyword(s):  
Total Knee Arthroplasty ◽  
Internal Consistency ◽  
Knee Arthroplasty ◽  
Convergent Validity ◽  
Interrater Reliability ◽  
Intraclass Correlation ◽  
Cronbach Alpha ◽  
Ceiling Effects ◽  
Total Knee ◽  
Abc Scale

BackgroundThe Balance Evaluation Systems Test (BESTest) is a relatively new balance assessment tool. Recently, the Mini-BESTest and the Brief-BESTest, which are shortened versions of the BESTest, were developed.ObjectiveThe purpose of this study was to estimate interrater and intrarater-interoccasion reliability, internal consistency, concurrent and convergent validity, and floor and ceiling effects of the 3 BESTests and other related measures, namely, the Berg Balance Scale (BBS), Functional Gait Assessment (FGA), and Activities-specific Balance Confidence (ABC) Scale, among patients with total knee arthroplasty (TKA).DesignThis was an observational measurement study.MethodsTo establish interrater reliability, the 3 BESTests were administered by 3 independent raters to 25 participants with TKA. Intrarater-interoccasion reliability was evaluated in 46 participants with TKA (including the 25 individuals who participated in the interrater reliability experiments) by repeating the 3 BESTests, BBS, and FGA within 1 week by the same rater. Internal consistency of each test also was assessed with Cronbach alpha. Validity was assessed in another 46 patients with TKA by correlating the 3 BESTests with BBS, FGA, and ABC. The floor and ceiling effects also were examined.ResultsThe 3 BESTests demonstrated excellent interrater reliability (intraclass correlation coefficient [ICC] [2,1]=.96–.99), intrarater-interoccasion reliability (ICC [2,1]=.92–.96), and internal consistency (Cronbach alpha=.96–.98). These values were comparable to those for the BBS and FGA. The 3 BESTests also showed moderate-to-strong correlations with the BBS, FGA, and ABC (r=.35–.81), thus demonstrating good concurrent and convergent validity. No significant floor and ceiling effects were observed, except for the BBS.LimitationsThe results are generalizable only to patients with TKA due to end-stage knee osteoarthritis.ConclusionsThe 3 BESTests have good reliability and validity for evaluating balance in people with TKA. The Brief-BESTest is the least time-consuming and may be more useful clinically.

scholarly journals Extended Version of a Test Battery for Visual Assessment of Postural Orientation Errors: Face Validity, Internal Consistency, and Reliability

2020 ◽  
Vol 100 (9) ◽  
pp. 1542-1556 ◽  
Author(s):  
Jenny Nae ◽  
Mark W Creaby ◽  
Eva Ageberg
Keyword(s):  
Internal Consistency ◽  
Interrater Reliability ◽  
Late Phase ◽  
Intraclass Correlation ◽  
Acl Injury ◽  
Visual Assessment ◽  
Test Battery ◽  
Face Validity ◽  
Extended Version ◽  
Postural Orientation

Abstract Objective Undesirable postural orientation may be a risk factor for a second anterior cruciate ligament (ACL) injury. The purpose of this study was to evaluate face validity, internal consistency, and interrater reliability of an extended version of a previous test battery for visual assessment of postural orientation errors (POEs) in patients during the late phase of rehabilitation following ACL reconstruction (ACLR) (ie, when they have initiated jumping exercises). Methods This study used a cross-sectional design. Fifty-three patients (45% women) in the late phase of ACLR rehabilitation performed 5 functional tasks of varying difficulty. POEs of the lower extremity and trunk were visually assessed from video and scored on a scale from 0 (good) to 2 (poor). Results The side-hop and 2 new POEs (femur medial to shank, femoral valgus) were added to the test battery after expert focus group discussions. Internal consistency was calculated for all tasks (α = .712–.823). Interrater reliability showed fair to substantial agreement for femur medial to shank and femoral valgus during all tasks (K = 0.31–0.815) and almost perfect agreement for side-hop (intraclass correlation coefficient = 0.88). Conclusions The good internal consistency and reliability after adding side-hop, femur medial to shank, and femoral valgus suggests that this test battery is a suitable tool to quantify postural orientation throughout ACLR rehabilitation. Impact This test battery for visual assessment of POEs was evaluated in a heterogeneous group of patients in different phases of ACLR battery and can be used in clinical practice to measure POEs in patients with ACLR, including in the late phase of rehabilitation to return to sport. This study encourages research on more demanding tasks and additional POEs to cover the entire rehabilitation period after ACL injury or reconstruction.

scholarly journals Development of an Activity Scale for Individuals With Advanced Parkinson Disease: Reliability and “On-Off” Variability

2000 ◽  
Vol 80 (11) ◽  
pp. 1087-1096 ◽  
Author(s):  
Alice Nieuwboer ◽  
Willy De Weerdt ◽  
René Dom ◽  
Kris Bogaerts ◽  
Godelieve Nuyens
Keyword(s):  
Parkinson Disease ◽  
Interrater Reliability ◽  
Intraclass Correlation ◽  
Correlation Coefficients ◽  
Weighted Kappa ◽  
Repeated Measurements ◽  
Functional Mobility ◽  
Variable Response ◽  
Phase Measurements ◽  
The Impact

Abstract Background and Purpose. Functional mobility in people with advanced Parkinson disease, some of whom have a variable response to drug treatment, is often difficult to evaluate. The objectives of this study were to investigate the interrater reliability of measurements obtained with a scale designed to measure mobility and to determine the impact of self-rated dyskinesias and fluctuations on the measure. Subjects. Twenty-nine people with Parkinson disease and with disability and considerable disease duration (X̄=11.7 years, SD=4.9, range=6–22) took part in the study. Methods. The subjects' performance on a 10-item scale was videotaped. The videotapes were then scored by 2 independent raters, and the scores were used to determine interrater reliability. The stability of 6 repeated measurements was examined in the home situation, taking into account self-rated fluctuations of motor performance. Results. Weighted Kappa values of agreement (.86–.98) confirmed the reliability between testers. Measurement during the “on” phase (when medication was working optimally) and the “off” phase (when the action of medication was strongly decreased or absent) led to different measurements. Measuring frequently within “on” and “off” phases gave relatively stable measurements for total function, bed transfers, and gait akinesia, the latter during the “off” phase only (intraclass correlation coefficients [ICCs]=.70–.93). However, more modest repeatability applied to transfers from a chair (ICC=.65–.67). Conclusion and Discussion. To ensure valid results in future effect studies, clinical differentiation between “on” and “off” phase measurements is proposed on the basis of patients' own perception of their medication status.

scholarly journals A Mobile App to Rapidly Appraise the In-Store Food Environment: Reliability, Utility, and Construct Validity Study (Preprint)

2019 ◽  
Author(s):  
Emma Joy McMahon ◽  
Rachael Jaenke ◽  
Julie Brimblecombe
Keyword(s):  
Construct Validity ◽  
Internal Consistency ◽  
Food Environment ◽  
Interrater Reliability ◽  
Intraclass Correlation ◽  
End Users ◽  
Mobile App ◽  
Convenience Stores ◽  
Significant Difference ◽  
Store Environment

BACKGROUND Consumer food environments are increasingly being recognized as influential determinants of food purchasing and subsequent intake and health. We developed a tool to enable efficient, but relatively comprehensive, appraisal of the in-store food environment. The Store Scout mobile app facilitates the evaluation of product (availability and range), placement (visibility, accessibility, proximity to high-traffic areas, and location relative to other products), price (price promotion), and promotion (displays and advertising) across 7 categories of food products, with appraisal given immediately as scores (0-100, where a higher score is more in line with best practice). Primary end users are public health nutritionists and nutritionists employed by store organizations; however, store managers and staff are also potential end users. OBJECTIVE This study aims to evaluate the reliability (interrater reliability and internal consistency), utility (distribution of scores), and construct validity (score by store type) of measurements using the Store Scout mobile app. METHODS The Store Scout mobile app was used independently by 2 surveyors to evaluate the store environment in 54 stores: 34 metropolitan stores (9 small and 11 large supermarkets, 10 convenience stores, and 4 petrol stations) in Brisbane, Australia, and 20 remote stores (19 small supermarkets and 1 petrol station) in Indigenous Australian communities in Northern Australia. The agreement between surveyors in the overall and category scores was evaluated using intraclass correlation coefficients (ICCs). Interrater reliability of measurement items was assessed using percentage agreement and the Gwet agreement coefficient (AC). Internal consistency was assessed by comparing the responses of items measuring similar aspects of the store environment. We examined the distribution of score values using boxplots and differences by store type using the Kruskal-Wallis test. RESULTS The median difference in the overall score between surveyors was 4.4 (range 0.0-11.1), with an ICC of 0.954 (95% CI 0.914-0.975). Most measurement items had very good (n=74/196, 37.8%) or good (n=81/196, 41.3%) interrater reliability using the Gwet AC. A minimal inconsistency of measurement was found. Overall scores ranged from 19.2 to 81.6. There was a significant difference in score by store type (<i>P</i>&lt;.001). Large Brisbane supermarkets scored highest (median 77.4, range 53.2-81.6), whereas small Brisbane supermarkets (median 63.9, range 41.0-71.3) and small remote supermarkets (median 63.8, range 56.5-74.9) scored significantly higher than Brisbane petrol stations (median 33.1, range 19.2-37.8) and convenience stores (median 39.0, range 22.4-63.8). CONCLUSIONS These findings suggest good reliability and internal consistency of food environment measurements using the Store Scout mobile app. We identified specific aspects that can be improved to further increase the reliability of this tool. We found a good distribution of score values and evidence that scoring could capture differences by store type in line with previous evidence, which gives an indication of construct validity. The Store Scout mobile app shows promise in its capability to measure and track the health-enabling characteristics of store environments.

Reliability and convergent validity of the Emotional Intelligence Admission Essay Scale, revised

2018 ◽  
Vol 81 (9) ◽  
pp. 543-550
Author(s):  
Sharon A Gutman ◽  
Janet P Falk-Kessler
Keyword(s):  
Emotional Intelligence ◽  
Behavior Problems ◽  
Internal Consistency ◽  
Behavioral Problems ◽  
Convergent Validity ◽  
Interrater Reliability ◽  
Intraclass Correlation ◽  
Professional Behavior ◽  
Scale Scores ◽  
High Intraclass Correlation Coefficient

Introduction The purpose of this study was to reassess the reliability and convergent validity of the revised Emotional Intelligence Admission Essay Scale, and determine whether the scale could identify students demonstrating professional behavior problems in the classroom and fieldwork environments. Method Thirty-six student participants completed the revised Emotional Intelligence Admission Essay Scale and Schutte Assessing Emotions Scale. Interrater reliability, internal consistency, and convergent validity were established. Results Interrater reliability and internal consistency were found to be high (intraclass correlation coefficient = .82, p < .001; Cronbach’s alpha = .96, p < .001, respectively). When participants were separated by age, convergent validity between the Emotional Intelligence Admission Essay Scale and Schutte Assessing Emotions Scale of participants aged ≥ 26 years was high ( rs = .83, p < .002) compared to those ≤ 25 years ( rs = .58, p < .002). Six participants (16.6%) received Emotional Intelligence Admission Essay Scale scores of 0 and were identified as potentially exhibiting professional behavioral problems; three of these students demonstrated professional behavior problems in the academic and/or fieldwork settings. Conclusion The Emotional Intelligence Admission Essay Scale can be used as a screen to identify whether students may exhibit professional behavior problems; however, caution should be used as some identified students may be able to prevent professional behavior problems once aware of program expectations.

scholarly journals A Multirater Instrument for the Assessment of Simulated Pediatric Crises

2011 ◽  
Vol 3 (1) ◽  
pp. 88-94 ◽  
Author(s):  
Aaron W Calhoun ◽  
Megan Boone ◽  
Karen H Miller ◽  
Rebecca L Taulbee ◽  
Vicki L Montgomery ◽  
...  
Keyword(s):  
Communication Skills ◽  
Internal Consistency ◽  
Interrater Reliability ◽  
Gap Analysis ◽  
Clinical Skills ◽  
Intraclass Correlation ◽  
Medical Knowledge ◽  
Self Reflection ◽  
Simulation Based ◽  
Self Assessments

Abstract Background Few validated instruments exist to measure pediatric code team skills. The goal of this study was to develop an instrument for the assessment of resuscitation competency and self-appraisal using multirater and gap analysis methodologies. Methods Multirater assessment with gap analysis is a robust methodology that enables the measurement of self-appraisal as well as competency, offering faculty the ability to provide enhanced feedback. The Team Performance during Simulated Crises Instrument (TPDSCI) was grounded in the Accreditation Council for Graduate Medical Education competencies. The instrument contains 5 competencies, each assessed by a series of descriptive rubrics. It was piloted during a series of simulation-based interdisciplinary pediatric crisis resource management education sessions. Course faculty assessed participants, who also did self-assessments. Internal consistency and interrater reliability were analyzed using Cronbach α and intraclass correlation (ICC) statistics. Gap analysis results were examined descriptively. Results Cronbach α for the instrument was between 0.72 and 0.69. The overall ICC was 0.82. ICC values for the medical knowledge, clinical skills, communication skills, and systems-based practice were between 0.87 and 0.72. The ICC for the professionalism domain was 0.22. Further examination of the professionalism competency revealed a positive skew, 43 simulated sessions (98%) had significant gaps for at least one of the competencies, 38 sessions (86%) had gaps indicating self-overappraisal, and 15 sessions (34%) had gaps indicating self-underappraisal. Conclusions The TPDSCI possesses good measures of internal consistency and interrater reliability with respect to medical knowledge, clinical skills, communication skills, systems-based practice, and overall competence in the context of simulated interdisciplinary pediatric medical crises. Professionalism remains difficult to assess. These results provide an encouraging first step toward instrument validation. Gap analysis reveals disparities between faculty and self-assessments that indicate inadequate participant self-reflection. Identifying self-overappraisal can facilitate focused interventions.

scholarly journals A Mobile App to Rapidly Appraise the In-Store Food Environment: Reliability, Utility, and Construct Validity Study

10.2196/16971 ◽  
2020 ◽  
Vol 8 (7) ◽  
pp. e16971 ◽  
Author(s):  
Emma Joy McMahon ◽  
Rachael Jaenke ◽  
Julie Brimblecombe
Keyword(s):  
Construct Validity ◽  
Internal Consistency ◽  
Food Environment ◽  
Interrater Reliability ◽  
Intraclass Correlation ◽  
End Users ◽  
Mobile App ◽  
Convenience Stores ◽  
Significant Difference ◽  
Store Environment

Background Consumer food environments are increasingly being recognized as influential determinants of food purchasing and subsequent intake and health. We developed a tool to enable efficient, but relatively comprehensive, appraisal of the in-store food environment. The Store Scout mobile app facilitates the evaluation of product (availability and range), placement (visibility, accessibility, proximity to high-traffic areas, and location relative to other products), price (price promotion), and promotion (displays and advertising) across 7 categories of food products, with appraisal given immediately as scores (0-100, where a higher score is more in line with best practice). Primary end users are public health nutritionists and nutritionists employed by store organizations; however, store managers and staff are also potential end users. Objective This study aims to evaluate the reliability (interrater reliability and internal consistency), utility (distribution of scores), and construct validity (score by store type) of measurements using the Store Scout mobile app. Methods The Store Scout mobile app was used independently by 2 surveyors to evaluate the store environment in 54 stores: 34 metropolitan stores (9 small and 11 large supermarkets, 10 convenience stores, and 4 petrol stations) in Brisbane, Australia, and 20 remote stores (19 small supermarkets and 1 petrol station) in Indigenous Australian communities in Northern Australia. The agreement between surveyors in the overall and category scores was evaluated using intraclass correlation coefficients (ICCs). Interrater reliability of measurement items was assessed using percentage agreement and the Gwet agreement coefficient (AC). Internal consistency was assessed by comparing the responses of items measuring similar aspects of the store environment. We examined the distribution of score values using boxplots and differences by store type using the Kruskal-Wallis test. Results The median difference in the overall score between surveyors was 4.4 (range 0.0-11.1), with an ICC of 0.954 (95% CI 0.914-0.975). Most measurement items had very good (n=74/196, 37.8%) or good (n=81/196, 41.3%) interrater reliability using the Gwet AC. A minimal inconsistency of measurement was found. Overall scores ranged from 19.2 to 81.6. There was a significant difference in score by store type (P<.001). Large Brisbane supermarkets scored highest (median 77.4, range 53.2-81.6), whereas small Brisbane supermarkets (median 63.9, range 41.0-71.3) and small remote supermarkets (median 63.8, range 56.5-74.9) scored significantly higher than Brisbane petrol stations (median 33.1, range 19.2-37.8) and convenience stores (median 39.0, range 22.4-63.8). Conclusions These findings suggest good reliability and internal consistency of food environment measurements using the Store Scout mobile app. We identified specific aspects that can be improved to further increase the reliability of this tool. We found a good distribution of score values and evidence that scoring could capture differences by store type in line with previous evidence, which gives an indication of construct validity. The Store Scout mobile app shows promise in its capability to measure and track the health-enabling characteristics of store environments.

scholarly journals High-Level Mobility Assessment Tool (HiMAT): Interrater Reliability, Retest Reliability, and Internal Consistency

2006 ◽  
Vol 86 (3) ◽  
pp. 395-400 ◽  
Author(s):  
Gavin P Williams ◽  
Kenneth M Greenwood ◽  
Val J Robertson ◽  
Patricia A Goldie ◽  
Meg E Morris
Keyword(s):  
Internal Consistency ◽  
Interrater Reliability ◽  
Assessment Tool ◽  
Physical Therapists ◽  
Intraclass Correlation ◽  
Systematic Change ◽  
Retest Reliability ◽  
Mobility Assessment ◽  
High Level ◽  
Very High

Abstract Background and Purpose. The High-Level Mobility Assessment Tool (HiMAT) assesses high-level mobility in people who have sustained a traumatic brain injury (TBI). The purpose of this study was to investigate the interrater reliability, retest reliability, and internal consistency of data obtained with the HiMAT. Subjects. Three physical therapists and 103 people with TBI were recruited from a rehabilitation hospital. Methods. Three physical therapists concurrently assessed a subset of 17 subjects with TBI to investigate interrater reliability. One physical therapist assessed a different subset of 20 subjects with TBI on 2 occasions, 2 days apart, to investigate retest reliability. Data from the entire sample of 103 subjects were used to investigate the internal consistency of this new scale. Results. Both the interrater reliability (intraclass correlation coefficient [ICC]=.99) and the retest reliability (ICC=.99) of the HiMAT data were very high. For retest reliability, a small systematic change was detected (t=3.82, df=19), indicating a marginal improvement of 1 point at retest. Internal consistency also was very high (Cronbach alpha=.97). Discussion and Conclusion. The HiMAT is a new tool specifically designed to measure high-level mobility, which currently is not a component of existing scales used in TBI. This study demonstrated that the HiMAT is a reliable tool for measuring high-level mobility. [Williams GP, Greenwood KM, Robertson VJ, et al. High-Level Mobility Assessment Tool (HiMAT): interrater reliability, retest reliability, and internal consistency. Phys Ther. 2006;86:395–400.]

Interobserver Reliability Using the Phonetic Level Evaluation With Severely and Profoundly Hearing-Impaired Children

1991 ◽  
Vol 34 (5) ◽  
pp. 989-999 ◽  
Author(s):  
Stephanie Shaw ◽  
Truman E. Coggins
Keyword(s):  
Interrater Reliability ◽  
Interobserver Reliability ◽  
Intraclass Correlation ◽  
Correlation Coefficients ◽  
Hearing Impaired ◽  
Intraclass Correlation Coefficients ◽  
Assessment Measure ◽  
Impaired Children ◽  
Speech Assessment ◽  
Hearing Impaired Children

This study examines whether observers reliably categorize selected speech production behaviors in hearing-impaired children. A group of experienced speech-language pathologists was trained to score the elicited imitations of 5 profoundly and 5 severely hearing-impaired subjects using the Phonetic Level Evaluation (Ling, 1976). Interrater reliability was calculated using intraclass correlation coefficients. Overall, the magnitude of the coefficients was found to be considerably below what would be accepted in published behavioral research. Failure to obtain acceptably high levels of reliability suggests that the Phonetic Level Evaluation may not yet be an accurate and objective speech assessment measure for hearing-impaired children.

Sign in / Sign up

Export Citation Format

Share Document