Psychometrics in Behavioral Software Engineering: A Methodological Introduction with Guidelines

A meaningful and deep understanding of the human aspects of software engineering (SE) requires psychological constructs to be considered. Psychology theory can facilitate the systematic and sound development as well as the adoption of instruments (e.g., psychological tests, questionnaires) to assess these constructs. In particular, to ensure high quality, the psychometric properties of instruments need evaluation. In this article, we provide an introduction to psychometric theory for the evaluation of measurement instruments for SE researchers. We present guidelines that enable using existing instruments and developing new ones adequately. We conducted a comprehensive review of the psychology literature framed by the Standards for Educational and Psychological Testing. We detail activities used when operationalizing new psychological constructs, such as item pooling, item review, pilot testing, item analysis, factor analysis, statistical property of items, reliability, validity, and fairness in testing and test bias. We provide an openly available example of a psychometric evaluation based on our guideline. We hope to encourage a culture change in SE research towards the adoption of established methods from psychology. To improve the quality of behavioral research in SE, studies focusing on introducing, validating, and then using psychometric instruments need to be more common.

Food Insecurity Measurement: Stakeholder Comparisons of the EU-SILC and HFSSM Indicators and Considerations Towards the Usefulness of a Headline Indicator

Prior to the February 2019 announcement that the Household Food Security Survey Module (HFSSM) will be used to estimate household food insecurity, there has not been a standardised measurement approach used in the United Kingdom (UK). Measurement has instead been somewhat inconsistent, and various indicators have been included in national and regional surveys. There remains a gap relating to the comparative usefulness of current and past food insecurity measures used in Northern Ireland (NI) (HFSSM; European Union-Survey of Income and Living Conditions (EU-SILC) food deprivation questions), and the potential usefulness of a headline indicator similar to that used to measure fuel poverty. This study presents findings from Northern Ireland (NI) stakeholder interviews (n = 19), which examined their perspectives on food insecurity measures which have previously been or are currently, or could potentially, be used in the UK/NI (HFSSM; EU-SILC food deprivation questions; headline indicator). Interview transcripts were coded using QSR NVivo (v.12) and inductively analysed to identify relevant themes. Stakeholders preferred the HFSSM to the EU-SILC, reasoning that it is more relevant to the food insecurity experience. A headline indicator for food insecurity was considered useful by some; however, there was consensus that it would not fully encapsulate the food insecurity experience, particularly the social exclusion element, and that it would be a complex measure to construct, with a high degree of error. This research endorses the use of the HFSSM to measure food insecurity in the UK, and provides recommendations for consideration of any future modification of the HFSSM or EU-SILC measurement instruments.

Critical Thinking and Effective Personality in the Framework of Education for Sustainable Development

According to UNESCO, critical thinking (CT) is a strategic skill for the 21st century, as it is associated with attitudes of personal and social change and improvement. Based on this, the objectives of this study were (1) to find out the possible relationships between CT and effective personality (EP) and (2) to determine the extent to which EP predicts the development of CT in children and adolescents. The research approach was quantitative, correlational and predictive. The sample consisted of 562 Spanish students. The measurement instruments were: (1) the Critical Thinking Disposition Inventory and (2) the Efficacy Personality Questionnaire. The main findings showed that there is a positive relationship between EP and CT, with the traits “being enthusiastic”, “developing positive self-esteem” and “having self-confidence” correlating most strongly with CT. The regression analysis shows that EP influences the development of students’ CT. There were no significant differences according to sex, and Primary Education students obtained higher CT scores. It is concluded that the better the promotion of EP, the greater the CT development will be. This suggests the need to design educational programmes for the improvement of EP, especially in adolescents.

The promise of computer adaptive testing in collection of orthopaedic outcomes: an evaluation of PROMIS utilization

Background A crucial component to improving patient care is better clinician understanding of patients’ health-related quality of life (HRQoL). In orthopaedic surgery, HRQoL assessment instruments such as the NIH developed Patient Reported Outcomes Measurement Information System (PROMIS), provide surgeons with a framework to assess how a treatment or medical condition is affecting each patient’s HRQoL. PROMIS has been demonstrated as a valuable instrument in many diseases; however, the extent to which orthopaedic surgery subspecialties have used and validated PROMIS measures in peer-reviewed research is unclear. Methods Systematic scoping methodology was used to investigate the characteristics of studies using PROMIS to assess HRQoL measures as orthopaedic surgical outcomes as well as studies validating computerized adaptive test (CAT) PROMIS physical health (PH) domains including: Physical Function (PF), Upper Extremity (UE), Lower Extremity (LE). Results A systematic search of PubMed identified 391 publications utilizing PROMIS in orthopaedics; 153 (39%) were PROMIS PH CAT validation publications. One-hundred publications were in Hand and Upper Extremity, 69 in Spine, 44 in Adult Reconstruction, 43 in Foot and Ankle, 43 in Sports, 37 in Trauma, 31 in General orthopaedics, and 24 in Tumor. From 2011 through 2020 there was an upward trend in orthopaedic PROMIS publications each year (range, 1–153) and an increase in studies investigating or utilizing PROMIS PH CAT domains (range, 1–105). Eighty-five percent (n = 130) of orthopaedic surgery PROMIS PH CAT validation publications (n = 153) analyzed PF; 30% (n = 46) analyzed UE; 3% (n = 4) analyzed LE. Conclusions PROMIS utilization within orthopaedics as a whole has significantly increased within the past decade, particularly within PROMIS CAT domains. The existing literature reviewed in this scoping study demonstrates that PROMIS PH CAT domains (PF, UE, and LE) are reliable, responsive, and interpretable in most contexts of patient care throughout all orthopaedic surgery subspecialties. The expanded use of PROMIS CATs in orthopaedic surgery highlights the potential for improved quality of patient care. While challenges of integrating PROMIS into electronic medical records exist, expanded use of PROMIS CAT measurement instruments throughout orthopaedic surgery should be performed. Plain english summary In orthopaedic surgery, health-related quality of life tools such as the NIH developed Patient Reported Outcomes Measurement Information System (PROMIS), offer patients an opportunity to better understand their medical condition and be involved in their own care. Additionally, PROMIS provides surgeons with a framework to assess how a treatment or medical condition is affecting each patient’s functional status and quality of life. The efficacy of PROMIS has been demonstrated in many diseases; however, its application throughout orthopaedic care has yet to be depicted. This study sought to identify the extent to which all orthopaedic surgery subspecialties have used and validated PROMIS measures in peer-reviewed research in order to identify its potential as an applicable and valuable tool across specialties. We determined that PROMIS utilization has significantly increased within the past decade. The existing literature reviewed in this scoping study demonstrates that the PROMIS computerized adaptive test domains evaluating physical function status are reliable, responsive, and interpretable in most contexts of patient care throughout all orthopaedic surgery subspecialties. Based on these results, this study recommends the expanded and more uniform use of PROMIS computerized adaptive test measurement instruments in the clinical care of orthopaedic patients.

Psychometric Properties of the Barthel Index Used at Intensive Care Unit Discharge

Background The Barthel Index, originally developed and validated to assess activities of daily living in patients with neuromuscular disorders, is commonly used in research and clinical practice involving critically ill patients. Objectives To evaluate the internal consistency, reliability, measurement error, and construct validity of the Barthel Index used at intensive care unit discharge. Methods In this observational study, 2 physiotherapists measured the physical functioning of 122 patients at intensive care unit discharge, using the Barthel Index and other measurement instruments. Results The patients had a median (IQR) age of 56 (47-66) years, and 62 patients (51%) were male. The primary reason for intensive care unit admission was sepsis (28 patients [23%]), and 83 patients (68%) were receiving mechanical ventilation. The Cronbach α value indicating internal consistency was 0.81. For interrater reliability, the intraclass correlation coefficient for the total score was 0.98 (95% CI, 0.97-0.98; P < .001) and the κ statistic for the individual items was 0.54 to 0.94. The standard error of measurement was 7.22, the smallest detectable change was 20.01, and the 95% limits of agreement were –10.3 and 11.8. The Barthel Index showed moderate to high correlations with the other physical functioning measurement instruments (ρ = 0.57 to 0.88; P < .001 for all). Conclusion The Barthel Index is a reliable and valid instrument for assessing physical functioning at intensive care unit discharge.

Digital Government as Implementation Means for Sustainable Development Goals

One of the challenges for implementing Sustainable Development Goals (SDGs) is the measurement of indicators that represent progress towards such goals. Measuring such progress enables data-driven decision-making and management of SDG-relevant projects and strategies. The premise of this research is that measuring such indicators depends on measuring so-called means of implementation, i.e. activities that directly contribute to the achievement of SDGs. Building on this premise, this article studies how the measurement of digital government (DG) can contribute to the measurement of SDGs. In particular, how the indicators originating in three DG measurement instruments can inform the SDG indicators. The main finding is an alignment matrix, showing how the DG indicators contribute with varying level of specificity to the measurement of 10 SDG indicators.

Anxiety in Angelman Syndrome

Angelman Syndrome (AS) is a neurodevelopmental disorder most commonly caused by the impaired expression of the maternal UBE3A gene on chromosome 15. Though anxiety has been identified as a frequently present characteristic in AS, there are limited studies examining anxiety in this population. Studies of anxiety in other neurodevelopmental disorders have found disorder specific symptoms of anxiety and age specific displays of anxiety symptoms. However, there is a consistent challenge in identifying anxiety in people with neurodevelopmental disorders given the lack of measurement instruments specifically designed for this population. Given the limited information about AS and anxiety, the aims of the current project were to (a) examine symptoms of anxiety in children with AS and (b) determine the correlates of anxiety in children with AS. Participants included 42 adult caregivers of youth with AS in the AS Natural History study who completed the Developmental Behavior Checklist (DBC). The results found that 26% of the sample demonstrated elevated symptoms of anxiety and established a relationship between elevated anxiety in youth with AS and higher levels of irritability, hyperactivity, self-absorbed behaviors, and disruptive/antisocial behaviors. Findings from this research provide a foundation for tailoring evidence-based assessments and treatments for youth with AS and anxiety.

The Effect of Learning Strategy and Writing Motivation on the Ability to Write Short Texts during Covid-19 Pandemic

The purpose of this study is to compare how students taught with Learning Strategies for Improving Thinking Skills (SPPKB), and expository learning strategies to write short stories. Additionally, it identifies differences in students' short story writing abilities between those with a high level of writing motivation and those with a low level of writing motivation. Additionally, it establishes the relationship between learning strategies and motivation for writing short stories in online learning. Diponegoro High School is located in the Asahan Regency. This research was conducted at Diponegoro Senior High School in Asahan Regency, with tenth-grade students as the research population. This is a quantitative study conducted using quasi-experimental methods. A Randomized Control Group Pretest-Posttest Design was used in the research. Instruments include treatment instruments such as syllabuses, lesson plans, questionnaires, and measurement instruments embedded in short story texts. The study's findings indicate a difference in the ability to write short stories between the treatment of learning strategies and motivation to write, and that this difference is mediated by an interaction.