Writing Open-ended Questions

I. GENERAL PRINCIPLES

Open-ended, essay or short answer questions?

• Open-ended questions differ from multiple-choice questions, because the student is not provided with options to choose from but must generate concepts and information.
• The literature makes further distinctions between essay questions (answer is several paragraphs) and short answer questions (a sentence or two)
• In the context of the Bridges curriculum, most exam questions will require answers of a few sentences to a couple of paragraphs long. To minimize confusion, we will refer to the questions as “open-ended”

Why Open-ended Questions?

• Well-designed open-ended questions promote deep learning, and allow assessment of analytical and critical thinking.
• Written questions offer students the opportunity to demonstrate their own judgment and vocabularies, and thereby reflect their “real-life” problem solving skills and application of knowledge more accurately than multiple-choice questions.
• Open-ended questions may promote different study habits: students are more likely to focus on broad issues, general concepts, and interrelationships.

Potential limitations:
Open-ended questions permit only a limited sampling of content learning due to the time required for students to respond – so pick the content carefully 
They can favor students who possess good writing skills – but they also promote good writing skills and allow students to practice clear writing
Students can go off on tangents or misunderstand the main point of the question – so ensure question is clear

II. GUIDELINES FOR WRITING OPEN-ENDED QUESTIONS AND EXAMS

The goal is to write questions that are worth students’ time and reflect important content, so that students will recognize and understand the value.

In the Bridges curriculum we use open-ended questions for

• Weekly required checkpoints for formative assessment (required weekly questions, 2 vignettes (cases or experiences), students choose one vignette, each vignette has 2-4 associated questions)
• Summative assessment (1-2 times per block, 5-6 vignettes with 2-5 questions each); total 14-16 questions on the exam.

Weekly formative checkpoint questions should incorporate all best practices and prepare students for the summative exam. Blocks should create a bank of questions that includes back-up questions for remediation and make-up exams, and ensure variation over the years to promote exam security. For the same reason, at least 2-3 new questions should be rotated in on an annual basis. New questions also help represent new and evolving content.

General best practices for open-ended questions:

• Questions should reflect the language that is used in the course materials. Test questions should be stated in simple, clear language.
• Proofread exam questions carefully and have another person proofread as well. 
• Weekly formative checkpoint exam questions should be of equal difficulty to those in the summative exam
• Do not use optional questions on summative exams– this affects reliability of grading and students may not study all the course material if they know they will have a choice.
• Be careful about having too many separate questions that depend upon answers or skills required in previous questions. A student’s initial mistake will be perpetuated over succeeding questions or tasks, penalizing the student repeatedly for one error.
• Keep questions free of nonfunctional material and extraneous clues.
• Test questions should also consider diversity and be free of race, ethnic, sex and other bias. They can be written to address issues such as race, ethnicity, and sexual orientation that affect health care and were taught in the course.

To create a high-quality exam, the questions in an exam should:

• Assess achievement of instructional learning objectives
• Measure important aspects of the subject (concepts and conceptual relations) in the context of authentic scenarios (patient cases, research scenarios, etc.)
• Accurately reflect the emphasis placed on important aspects of instruction
• Emphasize deep learning and application rather than recall
• Test on multiple disciplines in an integrated manner
• Have multiple sub-parts that test different discipline-based elements of the case

Bloom’s Taxonomy and Open-ended Questions:
Bloom’s taxonomy is a framework to categorize educational objectives and assessments according to complexity, with recall of facts at the bottom and more complex tasks such as analyzing and evaluating at the top.

Table 1: Bloom’s Taxonomy with verbs and sample question frames

Constructing Open-ended Questions:

The goal is to write open-ended questions that measure higher cognitive processes – apply, analyze, evaluate or create/synthesize.  

1. Questions will typically be based on a (patient) vignette, which should be concise and contain all relevant information for the learner to be able to answer the questions, without extraneous or redundant information.
2. In writing questions, carefully choose verbs that elicit the kind of thinking you want the students to demonstrate, based on Bloom’s taxonomy (Table 1).
   • Words like discuss and explain can be ambiguous. If you use explain or discuss, then give specific instructions as to what points should be explained or discussed.
   • Commonly used words like define and describe are associated with recall. o    You can either phrase questions as statements using any of the verbs listed, or as true questions using the type of wording listed under question frame.
   • Note that the table contains examples of question frames for all levels; but in the Bridges Curriculum we aim to use open-ended questions to assess application and above.
   • If you can answer questions without reading the vignette, they are likely recall only
3. Since we recognize that content knowledge is important, we encourage you to state explicitly what knowledge students need to demonstrate while they are providing answers to higher-level questions. For example: For example: “What medication would you prescribe to the patient in this vignette? Explain your answer [application] and include a description of the mechanism of action of this drug in your answer [content knowledge]”
4. Create a rubric as you write the question (see pages 5-6) it helps ensure you are truly assessing application of knowledge in addition to content knowledge. You don’t have to think of all the possible answers upfront, but outlining the rubric at time of question writing saves time and confusion with grading later
5. Formative OEQs will not be graded, but should prepare students for Summative OEQs so should really be very similar. So think about what a rubric might look like even when writing formative OEQs.
6. For each question, be sure to link the following:
   • Learning objective(s)
   • Disciplines addressed
   • Cognitive skill level (Bloom’s level)

III. RUBRIC PRINCIPLES AND GUIDELINES:

• Rubrics are often classified as holistic (to give an overall sense of student performance) versus analytic (to detail where students perform well). The UCSF Bridges rubrics have elements of both, but are mostly holistic. This combination allows for specific feedback to students, but also allows for flexibility for the block directors to decide what constitutes a good answer.
• Open-ended questions in the Bridges curriculum aim to test higher levels of Bloom Taxonomy objectives (apply, analyze, evaluate etc), but we recognize that content knowledge is important. The highest score can only be obtained if an answer is correct both in application and content knowledge. If a student only demonstrates content knowledge this will result in a lower score than demonstration of higher levels of ability according to Bloom’s taxonomy.
• Questions need to be constructed in such a way that the student knows to demonstrate both the content knowledge and application of knowledge
• The rubric divides the scores in three columns, with the highest scores for demonstration of application and other higher level cognitive processes and adequate content knowledge, middle scores for demonstration of application with limited or absent content knowledge OR for demonstration of application and knowledge but with significant errors in the answer, and the lowest scores if content knowledge only is provided. An item that remains unanswered or with an answer that is so limited that the quality cannot be judged will receive a score of 0
• Within each column, there are two score options (1-2, 3-4, 5-6). How to distinguish between these two options will vary by test item, and may not become apparent until you have some student answers to compare. In general, a score of 3 should be given to an answer in which the application is correct, but this seems to be based on guess-work or there are major limitations in the reasoning and/or content knowledge. A score of 4 would be appropriate if the ultimate answer is incorrect due to a misstep along the way, but content knowledge is overall adequate and the reasoning is adequate. To ensure transparency and clarity for the students it is however essential that faculty score answers similarly, and come to an agreement of what constitutes a higher versus a lower score.
• Since not all exams will have the same number of total questions, the total score across questions will be normalized to 100
• In creating the rubric for a specific question, it is good to have a model answer in mind, but the rubric is most useful if it contains the essential elements for each answer category, rather than all potential answer options. Constructing a rubric while creating the question helps ensure that the questions is complete and at the right Bloom’s level.
• In addition to knowledge and analytic thinking, students will be assessed on writing skills, contributing to max 1/6 of the total score. Anchors to help with rating writing skills are included in the rubric, and should be used to adjust the score on each item. If the score for answer content does not align with the corresponding description of writing style, content score should be decreased by 1 point. See the rubric template and example rubrics for further explanation.

Example Questions with Rubric Adapted from ABC

Disciplines: Physiology, Medicine | System: Pulmonary

Example 1. A 58 year-old man presents with hypoxemia 1 week after a myocardial infarction. Vital signs are: BP 75/55, HR 120, RR 22. Physical examination reveals a III/VI holosystolic murmur and clear lungs. A chest x-ray is normal. The following values are obtained: Hemoglobin: 15 grams/ 100 ml blood ABG drawn on 100% Oxygen: pH 7.39/PaCO2 30 mmHg/PaO2 50/ Saturation 85%. A Pulmonary Arterial catheter is inserted and mixed venous blood is obtained: Mixed venous blood: PmvO2 = 35 with a saturation of 68%.

A. Early learner:

Calculate the approximate percentage of shunt fraction:

• Provide the formula used in your calculation (knowledge)
• Calculate the shunt fraction (apply)

Learning objective: Define shunt and apply the shunt equation

B. More advanced learner:
You suspect the patient may have a VSD as explanation of his findings. Provide a rationale for this diagnosis based on the both the symptoms and the numbers provided, using appropriate formulas to support your argumentation with calculations as needed.
Learning objectives: Define shunt and apply the shunt equation; Define respiratory failure and distinguish between acute and chronic respiratory failure

A 65 year-old man with emphysema is intubated and place on a mechanical ventilator for respiratory failure. Now, 2 days later, he has the following values 

pH: 7.32 / PaCO2: 50 mmHg / PaO2: 110 mmHg. On ventilator settings of Rate: 12 breaths/minute / TV: 400 ml / FiO2: 50%
A. Early learner
Identify the acid base abnormality and explain your rationale, comparing the provided values for pH and pCO2 with normal values

• Provide normal values for pH and pCO2 (knowledge) and the terminology used for deviations in pH with definitions
• Describe the acid base abnormality in this case and explain your rationale (apply)

B. More advanced learner:
How do these numbers help you decide about the patient’s adequacy of gas exchange? Provide your opinion of this patient’s gas exchange, supporting it with a description of the patient’s acid-base status and A-a gradient as compared to normal, providing the values for the numbers that you base your assessment on.