The American College of Surgeons Entering Resident Readiness Assessment (ACS ERRA): Validity Evidence from a Three-Year National Study.

OBJECTIVE

To gather validity evidence supporting the use and interpretation of scores from the American College of Surgeons Entering Resident Readiness Assessment (ACS ERRA) Program.

SUMMARY AND BACKGROUND DATA

ACS ERRA is an online formative assessment program developed to assess entering surgery residents' ability to make critical clinical decisions, and includes 12 clinical areas and 20 topics identified by a national panel of surgeon educators and residency program directors.

METHODS

Data from three national testing administrations of ACS ERRA (2018-20) were used to gather validity evidence regarding content, response process, internal structure (reliability), relations to other variables, and consequences.

RESULTS

Over the three administrations, 1,975 surgery residents participated from 125 distinct residency programs. Overall scores (Mean = 64% [SD = 7%]) remained consistent across the three years (p = .670). There were no significant differences among resident characteristics (gender, age, IMG status). The mean case discrimination index was 0.54 [SD = .15]. Kappa inter-rater reliability for scoring was 0.87; the overall test score reliability (G-coefficient) was 0.86 (Φ-coefficient = 0.83). Residents who completed residency readiness programs had higher ACS ERRA scores (66% vs. 63%, Cohen's d = 0.23, p  .001). On average, 15% of decisions made (21/140 per test) involved potentially harmful actions. Variability in scores from graduating medical schools (7%) carried over twice as much weight than from matched residency programs (3%).

CONCLUSIONS

ACS ERRA scores provide valuable information to entering surgery residents and surgery program directors to aid in development of individual and group learning plans.