In higher education, an ideal way to evaluate higher order thinking (HOT) skill mastery in a learning environment is by a long, open form of communication – like essay questions, white papers, or perhaps even videos and images. The reality is that this is not always possible or feasible or cost effective in a corporate learning environment. Multiple choice questions, on the other hand, are quick and easy to score and tools are readily available. While they are commonly used for the superficial recall of knowledge, done right these questions can be an effective tool for assessing a manager's comprehension and critical thinking skills in a real-world context.
Pulling in questions from work-related examples and real world scenarios.
Ideally, management training is applied on the job as situations arise that require it. One of the most effective ways to assess a manager's comprehension is to use scenario-based questions that simulate real work experiences or actual work-related challenges that are likely to occur. Here is an example of a question stems that demonstrates how to upgrade a question from the recall to a higher-order thinking level using scenarios from Malamed (2021b) –
- Before: What is the first concern of an emergency worker?
- After: An emergency technician (ET) arrived at the scene of an accident. Three people were injured. One person had scrapes and bruises, a second was moaning in pain and a third was quickly losing blood from a skull injury. Why did the ET triage the skull injury as the highest priority?
Here is a sample vignette that requires learners to recall and synthesize information from Scully (2017).
A 28-year-old primigravid woman is at 11 weeks’ gestation. Medical history is unremarkable. Family history is unremarkable except that both of her brothers have intellectual developmental disorder, her mother died of breast cancer at age 55, and her father is estranged. No family health records are available.
Which of the following studies is appropriate?
- Amniocentesis for α-fetoprotein
- Blood test for fragile X carrier status
- Blood test for phenylketonuria carrier status
- Chorionic villus sampling for chromosome analysis
- Chorionic villus sampling for Duchenne’s muscular dystrophy
“Because of rapidly rising national defense expenditures, it is anticipated that Country A will experience a price inflation unless measures are taken to restrict the growth of aggregate private demand. Specifically, the government is considering either (1) increasing personal income tax rates or (2) introducing a very tight monetary policy.” If the government of Country A wishes to minimize the adverse effect of its anti-inflationary policies on economic growth, which one of the following policies should it use?
a. The tight money policy because it restricts consumption expenditures more than investment.
b. The tight money policy, since the tax increase would restrict consumption expenditures.
c. The personal income tax increase since it restricts consumption expenditures more than investment.
d. Either the tight money policy or the personal income tax rate increase since both depress investment equally.”
Drawing conclusions from visuals like job aids and charts
Create multiple choice questions that require the learner to first interpret this information before they answer the question. For example, present a graph or ask them what actions they would take based on the data or present the job aid and ask them to use it to make a judgement call based on available information.
Refer to this question stem from Malamed (2021b) below.
- Before: What is/are the function(s) of a computer’s projecting mechanism? (Select all that apply.)
- After: Using the repair flowchart shown here, what should you check if the monitor stops working? <– use job aids to answer a question.
Without other data, which conclusion can you make from reviewing Figure 17?
a. The average American uses more drugs than citizens of any country except the United Kingdom
b. The average Mexican or Chilean consumes fewer drugs than citizens from other countries.
c. Americans are more likely than residents of other countries to use new drugs.
d. Japanese have regulations that make it very difficult to obtain new drugs
Formulate questions with the help of Instructional Design frameworks
A third approach is to use Instructional Design models to formulate higher level multiple choice questions. These techniques are frequently used in educational institutions to help aid in the development of critical thinking skills in children. And, in cases where multiple choice questions are primarily knowledge based or need to be systematically applied to large populations, these techniques provide a plethora of ideas and insights. Question stems based on Bloom's Taxonomy verbs are designed to target learning experiences ranging from remembering to creating. Webb's Depth of Knowledge (DOK) is a framework that encourages learners to extend their understanding to new situations. These guidelines may be helpful. Paul's Elements of Reasoning may also provide useful insights on how to develop questions to build a learner's ability to apply logical and fact-based reasoning to new situations.
Here is an example of leveling up a multiple choice question from a recall question to one that requires comprehension using Bloom's Taxonomy (Assessing Higher-Order Outcomes Using Multiple-Choice Questions, 2021).
With well written multiple choice, you can go beyond this by asking learners to interpret facts, evaluate situations, explain cause and effect, make inferences, and predict results.
Sources:
Assessing Higher-Order Outcomes using Multiple-Choice Questions. (2021). Office of Teaching and Learning. https://otl.uoguelph.ca/teaching-learning-resources/assessment-student-learning/assessing-higher-order-outcomes-using
Blended Learning Resources. (2019, August 29). Landing – Developing multiple choice questions that test higher order thinking. http://blendedlearningresources.co.za/landing-developing-multiple-choice-questions-that-test-higher-order-thinking/#toggle-id-2
Brookhart, S. M. (2010). How to Assess Higher-Order Thinking Skills in Your Classroom (1st ed.). ASCD.
Malamed, C. (2021a, November 6). Writing Multiple Choice Questions For Higher Order Thinking. The eLearning Coach. https://theelearningcoach.com/elearning_design/higher-order-multiple-choice-questions/
Malamed, C. (2021b, November 14). 10 Rules For Writing Multiple Choice Questions. The eLearning Coach. https://theelearningcoach.com/elearning_design/rules-for-multiple-choice-questions/
Scully, D. (2017) “Constructing Multiple-Choice Items to Measure Higher-Order Thinking,” Practical Assessment, Research, and Evaluation: Vol. 22, Article 4. DOI: https://doi.org/10.7275/swgt-rj52
Available at: https://scholarworks.umass.edu/pare/vol22/iss1/4
Weimer, M., PhD. (2019, November 25). Multiple-Choice Tests: Revisiting the Pros and Cons. Faculty Focus | Higher Ed Teaching & Learning. https://www.facultyfocus.com/articles/educational-assessment/multiple-choice-tests-pros-cons/
Author Information
Felicia Littlejohn, Ph.D., is Learning and Development (L&D) professional, leadership coach, and former commercial business leader who empowers others to achieve their goals. Her work in leadership development, coaching, and strategy consulting has helped businesses, teams, and professionals to upskill, cultivate leadership potential, enhance performance, and overcome growth hurdles. Felicia is a Certified Professional Coach (C.P.C.) who holds a Ph.D. in Engineering, an M.B.A. in Marketing, and is currently pursuing an M.Ed. in Instructional Design.