Examining the Factors Influencing Applicants’ Placement on One General Surgery Program’s Rank Order List

Purpose Surgery residency applications include variables that determine an individual’s rank on a program’s match list. We performed this study to determine which residency application variables are the most impactful in creating our program’s rank order list. Methods We completed a retrospective examination of all interviewed applicants for the 2019 match. We recorded United States Medical Licensing Examinations (USMLE) step I and II scores, class quartile rank from the Medical Student Performance Evaluation (MSPE), Alpha Omega Alpha (AOA) membership, geographic region, surgery clerkship grade, and grades on other clerkships. The MSPE and letters of recommendation were reviewed by two of the authors and assigned a score of 1 to 3, where 1 was weak and 3 was strong. The same two authors reviewed the assessments from each applicant’s interview and assigned a score from 1-5, where 1 was poor and 5 was excellent. Univariate analysis was performed, and the significant variables were used to construct an adjusted multivariate model with significance measured at P < .05. Results Univariate analysis for all 92 interviewed applicants demonstrated that USMLE step 2 scores ( P = .002), class quartile rank ( P = .004), AOA status ( P = .014), geographic location ( P < .001), letters of recommendation ( P < .001), and interview rating ( P < .001) were significant in predicting an applicant’s position on the rank list. On multivariate analysis only USMLE step 2 ( P = .018) and interview ( P < .001) remained significant. Conclusion USMLE step 2 and an excellent interview were the most important factors in constructing our rank order list. Applicants with a demonstrated strong clinical fund of knowledge that develop a rapport with our faculty and residents receive the highest level of consideration for our program.

Potential Impact of Pass/Fail Scores on USMLE Step 1: Predictors of Excellence in Obstetrics and Gynecology Residency Training

AIM The study aims to determine resident applicant metrics most predictive of academic and clinical performance as measured by the Council of Resident Education in Obstetrics and Gynecology (CREOG) examination scores and Accreditation Council for Graduate Medical Education (ACGME) clinical performance (Milestones) in the aftermath of United States Medical Licensing Examination Scores (USMLE) Step 1 becoming a pass/fail examination. METHODS In this retrospective study, electronic and paper documents for Wayne State University Obstetrics and Gynecology residents matriculated over a 5-year period ending July 2018 were collected. USMLE scores, clerkship grade, and wording on the letters of recommendation as well as Medical Student Performance Evaluation (MSPE) were extracted from the Electronic Residency Application Service (ERAS) and scored numerically. Semiannual Milestone evaluations and yearly CREOG scores were used as a marker of resident performance. Statistical analysis on residents (n = 75) was performed using R and SPSS and significance was set at P < .05. RESULTS Mean USMLE score correlated with CREOG performance and, of all 3 Steps, Step 1 had the tightest association. MSPE and class percentile also correlated with CREOGs. Clerkship grade and recommendation letters had no correlation with resident performance. Of all metrics provided by ERAS, none taken alone, were as useful as Step 1 scores at predicting performance in residency. Regression modeling demonstrated that the combination of Step 2 scores with MSPE wording restored the predictive ability lost by Step 1. CONCLUSIONS The change of USMLE Step 1 to pass/fail may alter resident selection strategies. Other objective markers are needed in order to evaluate an applicant’s future performance in residency.

Grade Inflation in the Family Medicine Clerkship

Background and Objectives: Medical educators perceive grade inflation to be a serious problem. There is some literature discussing the magnitude of the problem and ways to remediate it, but little literature is available in the field of family medicine. We sought to examine what methods of remediating grade inflation have been tried by family medicine clerkship directors, and what factors influence the chosen method of addressing this problem. Methods: We conducted a national Council of Academic Family Medicine’s (CAFM) Educational Research Alliance (CERA) survey of family medicine clerkship directors, inquiring about their perceptions of the seriousness of grade inflation, whether it was perceived as a remediable problem, and what methods had been tried within the last 3 years to address this problem. Results: The response rate was 69%. Clerkship directors’ perceptions that grade inflation is a serious problem either nationally or in their own clerkship did not correlate with how they weighted the objective versus subjective portions of the clerkship grade. Clerkship directors who agreed that grade inflation was a remediable problem had a higher percentage of nonexamination objective criteria and a lower percentage of subjective criteria in their grading formula. Clerkship directors who agreed grade inflation is a problem in their clerkship were more likely to have tried giving feedback to graders on grade distribution than those who didn’t think grade inflation was a problem. Conclusions: Family medicine clerkship directors perceive grade inflation to be a serious problem, both at a national level and in their clerkships. Various methods of addressing grade inflation have been tried by family medicine clerkship directors.

Emergency Medicine Standardized Letter of Evaluation (SLOE): Findings From the New Electronic SLOE Format

ABSTRACT Background Emergency medicine (EM) uses a standardized template for residency application letters of recommendations. In 1997, the Standardized Letter of Recommendation was developed with categories for applicant comparison. Now named the Standardized Letter of Evaluation (SLOE), it is a universal requirement in the EM residency application process. In 2016, a website called “eSLOE” was launched for credentialed academic EM faculty to enter applicant data, which generates a SLOE. Objective This article outlines website creation for the eSLOE and its successful national implementation in the 2016–2017 EM residency application cycle. We analyzed current trends in applicant assessments from the eSLOE data and compared them to prior data. Methods Data from 2016–2017 were sorted and analyzed for each question on the eSLOE. An analysis of Global Assessment and Qualifications for EM rankings, clerkship grade, and comparison with prior SLOE data was performed. Results Analysis of 6715 eSLOEs for 3138 unique applicants revealed the following Global Assessment rankings: top 10%, top one-third, middle one-third, and lower one-third. There was less spread with the distribution for clerkship grade and Qualifications for EM. The 2011–2012 standard letter of recommendation global assessment data, with top 10%, top one-third, middle one-third, and lower one-third, also revealed top-clustered results with less spread compared with the ranking usage in 2016–2017. Conclusions Results indicate an improved spread of all rank categories for Global Assessment, enhancing the eSLOE's applicant discrimination. There has been an overall improvement in rank designation when compared with previously published data.

The Reliability of 2-Station Clerkship Objective Structured Clinical Examinations in Isolation and in Aggregate

Background: Most medical schools in the United States report having a 5- to 10-station objective structured clinical examination (OSCE) at the end of the core clerkship phase of the curriculum to assess clinical skills. We set out to investigate an alternative OSCE structure in which each clerkship has a 2-station OSCE. This study looked to determine the reliability of clerkship OSCEs in isolation to inform composite clerkship grading, as well as the reliability in aggregate, as a potential alternative to an end-of-third-year examination. Design: Clerkship OSCE data from the 2017-2018 academic year were analyzed: the generalizability coefficient (ρ2) and index of dependability (φ) were calculated for clerkships in isolation and in aggregate using variance components analysis. Results: In all, 93 students completed all examinations. The average generalizability coefficient for the individual clerkships was .47. Most often, the largest variance component was the interaction between the student and the station, indicating inconsistency in the performance of students between the 2 stations. Aggregate clerkship OSCE analysis demonstrated good reliability for consistency (ρ2 = .80). About one-third (33.8%) of the variance can be attributed to students, 8.2% can be attributed to the student by clerkship interaction, and 42.6% can be attributed to the student by block interaction, indicating that students’ relative performances varied by block. Conclusions: Two-station clerkship OSCEs have poor to fair reliability, and this should inform the weighting of the composite clerkship grade. Aggregating data results in good reliability. The largest source of variance in the aggregate was student by block, suggesting testing over several blocks may have advantages compared with a single day examination.