Assessment is a Compromise

This year’s visiting Presidential Chair, Dr. Cees van der Vleuten, delivered his keynote address on programmatic assessment. All assessment mechanisms involve necessary trade-offs of reliability, validity, educational impact, acceptability, and cost. Our mission as educators is not to develop an assessment that can be perfect in every way, but rather to make informed decisions about what to compromise on, and when.

Reliability, for example, is somewhat of a myth—no method is perfectly reliable—but we can statistically improve the reliability of our methods by using multiple data points for each method (never just one observation of a learner, but multiple observations, preferably across multiple raters). Even very subject measures can be quite robust (decreasing fears around bias/non-standardization) if we have enough data points for a learner when making a summative judgment.

The validity, too, is challenging to maximize in just one go; instead, we need to focus on using multiple assessment methods to give us a fuller picture of each learner, thereby optimizing the validity of the overall assessment. Because workplace-based assessment is fraught with issues, Dr. van der Vleuten’s work has revealed that the instrument is far less important than the person using it; we are better off focusing on training supervisors who assess learners than perfecting our assessment tools.

This work has led to the concept of programmatic assessment, which emphasizes the use of frequent formative assessment (feedback dialogue) and infrequent high-stakes summative assessment that incorporates lots of data points from multiple assessors and multiple methods. He suggests ceasing the use of individual assessments to make pass/fail decisions (which should come only from combined and more holistic assessments); instead, those high-stakes decisions should come via a team effort from a trained committee. Only through this approach, Dr. van der Vleuten explains, can we increase the “pixels” on our image of each learner, allowing us to get a crisp picture of their performance and accurately reflect that in our decisions.

Access the plenary session and keynote address (advance to 18.08 mark)