Background image

Medical Education
Graduate Medical Education

Professionalism

360 evaluations include observations by a variety of individuals from the multiple contexts where professionalism and communication skills are demonstrated. Ratings by self, peers, colleagues and faculty are generally collected using different tools than ratings by patients. The 360 method captures information on most of the competencies listed by the ACGME under “Professionalism” and “Interpersonal & Communication Skills and is highly recommended for every program’ assessment system.

Core Measure for UCSF GME

For Health Care Team and Self-Evaluation, we recommend the Professional Associate Evaluation Form, developed by the CREOG (Ob/Gyn) Competency Task Force

For Patient Surveys, we recommend the American Board of Internal Medicine (ABIM) Patient Survey

 

Professionalism - Health Care Team Evaluations 

 

Reliability and Validity:

The optimal number and frequency 360 assessments is uncertain. A feasible minimum to achieve inter-rater reliability = .80 may be 5 non-clinical and 6 clinical raters each on 2 occasions (1).

Preliminary evidence of construct validity shows modest growth in 360 scores comparing senior vs. junior residents with a magnitude similar to the growth in other competencies including critical self-reflection skills (2).

Content validity exists to the extent that the survey items actually assess the professionalism and communication skills they are intended to measure. The 9-item evaluation includes communication (patients/families, nursing/allied staff), respect (patients, nursing/allied staff), compassion, reliability, honesty/integrity, responsibility, and advocacy.

Administration

  • Timing: More frequent observations tied to specific learning experiences provide more valid data than the minimum, i.e., global impressions provided twice annually.
  • Who Performs: Many individuals can legitimately contribute to 360 assessments. The list usually includes faculty, residents (supervisors, peers and juniors), nursing and other clinical staff, and consultants. Selection should be based upon the individual’s opportunities to directly observe the resident interacting with patients and the health care team. N.B. Faculty need to complete the 360 evaluation separately from their global ratings at the end of the rotation so the data can be summarized and reported accurately.
  • Format: 9 items sample aspects of professionalism and communication skills. Each item is scored on 9-point scales with 1-3 = unsatisfactory, 4-6 = meets expectations, and 7-9=excellent.
  • Scoring Criteria and Training: Each item defines a specific trait in clear language. Standard-setting would be helpful for consistently distinguishing the 3 levels of unsatisfactory, the 3 levels of satisfactory and the 3 levels of excellent. Documentation: Assessment can be documented on E-value and learners can access the results confidentially.

Uses of the Data

  • Summarizing the data: Score averages, ranges, and comparative data for the PGY year are provided as part of the data report residents review before their semi-annual meetings. The other data sources for the 360 assessment include the relevant items (e.g., respect) from the residents’ clinical educator evaluations and patient surveys. These are also summarized as means and ranges compared with averages for the PGY year.
  • Formative Uses: Assessment results support development of professionalism and communication skills by individual residents, identify trends in performance across PGY groups and spur possible improvements to the curriculum in these two competencies
  • Summative Decisions and Remediation: Scores in the unsatisfactory (scores 1-3) would trigger remediation and a low satisfactory score (4) would trigger suggestions for improvement

Workflow Procedures

A systematic approach is recommended to maximize the use of the assessments and facilitate data management. An example follows in the Appendix.

References

1. Murphy DJ, Bruce DA, Mercer SW, Eva KW. The reliability of workplace-based assessment in postgraduate medical education and training: a national evaluation in general practice in the United Kingdom. Adv in Health Sci Educ 2008 DOI 10.1007/s10459-008-9104-8.

2. Learman LA, Autry AM, O'Sullivan P. Reliability and validity of reflection exercises for obstetrics and gynecology residents. Am J Obstet Gynecol 2008;198(4):461.e1-8; discussion 461.e8-10.360 evaluations include observations by a variety of individuals from the multiple contexts where professionalism and communication skills are demonstrated. Ratings by self, peers, colleagues and faculty are generally collected using different tools than ratings by patients. The 360 method captures information on most of the competencies listed by the ACGME under “Professionalism” and “Interpersonal & Communication Skills and is highly recommended for every program’ assessment system.

 

Professionalism - Patient Surveys 

 

Reliability and Validity

The optimal number of patient surveys is uncertain. As initially used in an ABIM continuing professional development context, 20 surveys were recommended. A feasibility study with Canadian internal medicine residents showed 12 outpatient surveys to be associated with poor reliability (0.56) (1).
Data using the Consultation and Relational Empathy (CARE) survey, which is used for physician accreditation in Scotland, suggest that more than 40 patients may be necessary for good inter-rater reliability (.80) with 25 patients providing adequate reliability (.70) (2).

Content validity exists to the extent that a measure actually assesses the communication and professionalism skills it is intended to measure. The ABIM’s 10 items sample multiple aspects of doctor-patient communication (greeting, listening, establishing rapport, explaining, inviting participation in decision-making) and professionalism (truthfulness, respect, sensitivity to linguistic barriers). The physician characteristics evaluated using the CARE survey are similar to those assessed using the ABIM survey. Although the CARE survey has some advantages in how it describes the characteristics being evaluated, we recommend the ABIM survey because it is so widely used and studied in the United States.

Administration

  • Timing: Patient satisfaction surveys may be obtained regularly as a quality measure. Otherwise, administration twice annually is the minimum recommended for assessment of competency.
  • Who Performs: Patients under the direct care of the resident.
  • Format: The ABIM Patient Survey includes 10 complex items. Both sample similar aspects of communication and professionalism skills.
  • Scoring Criteria and Training: The ABIM survey uses scales ranging from 1-5 (poor, fair, good, very good, and excellent), uses simple language, and relies upon the subjective experience of individual patients. Patients receive a general orientation but no specific instructions regarding the scoring criteria. Although a 9-point scale is used for the 360’s Health Care Team and Self-Evaluations, the patient surveys use only 5 points because finer distinctions are challenging for patients. This 5- vs. 9-point scale difference is important to note when scores are summarized and discussed with residents.
  • Documentation: Summaries with comparative data are made available for review at the semi-annual meetings.

Uses of the Data

 

  • Summarizing the data: Score averages, ranges, and comparative data for the PGY year are provided as part of the data report residents review before their semi-annual meetings. The other data sources for the 360 assessment include the relevant items (e.g., respect) from the residents’ clinical educator evaluations and patient surveys. These are also summarized as means and ranges compared with averages for the PGY year.
  • Formative Uses: There are many useful ways to use the assessment results to support development of professionalism and communication skills by individual residents, identify trends in performance across PGY groups and spur possible improvements to the curriculum.
  • Summative Decisions and Remediation: It is important to set criteria that would trigger a plan of improvement to explain the criteria to the residents before they are assessed.

Workflow Procedures

A systematic approach is recommended to maximize the use of the assessments and facilitate data management. There is an example in the Appendix.

References

1. Tamblyn R, Benaroya S, Snell L, McLeod P, Schnarch B, Abrahamowicz M. The feasibility and value of using patient satisfaction ratings to evaluate internal medicine residents. J Gen Intern Med 1994;9(3):146-52.
2. Murphy DJ, Bruce DA, Mercer SW, Eva KW. The reliability of workplace-based assessment in postgraduate medical education and training: a national evaluation in general practice in the United Kingdom. Adv in Health Sci Educ 2008 DOI 10.1007/s10459-008-9104-8.

 
 

 

Tags: