{"id":1453,"date":"2022-04-15T07:42:49","date_gmt":"2022-04-15T07:42:49","guid":{"rendered":"http:\/\/felicia.premierjones.com\/?p=1453"},"modified":"2022-05-05T21:55:01","modified_gmt":"2022-05-05T21:55:01","slug":"3-tips-for-assessing-management-skills-with-multiple-choice","status":"publish","type":"post","link":"https:\/\/premierjones.com\/felicia\/3-tips-for-assessing-management-skills-with-multiple-choice\/","title":{"rendered":"3 Tips for Assessing Management Skills with Multiple Choice"},"content":{"rendered":"\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

This post is part of a study on\u00a0How to Design Online Training Courses to Promote In Demand Strategic Thinking Skills in Adult Learners<\/em><\/strong>.\u00a0 The purpose of this study is to investigate ways to bring proven insights from the field of Instructional Design into the applied and practical world of Corporate Training.\u00a0 If you are interested in\u00a0 the science of learning, read on.\u00a0 If you are looking for how training can help your career and business, please check out my other post categories.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

In higher education, an ideal way to evaluate higher order thinking (HOT) skill mastery in a learning environment is by a long, open form of communication – like essay questions, white papers, or perhaps even videos and images.\u00a0 The reality is that this is not always possible or feasible or cost effective in a corporate learning environment.\u00a0 Multiple choice questions, on the other hand, are quick and easy to score and tools are readily available.\u00a0 While they are commonly used for the superficial recall of knowledge, done right these questions can be an effective tool for assessing a manager’s comprehension and critical thinking skills in a real-world context.\u00a0<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

\n\t\t\t\t\t\t
\n\t\t\t\t\t
\n\t\t\t
\n\t\t\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t

Pulling in questions from work-related examples and real world scenarios.<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
\n\t\t\t\t
\n\t\t\t\t\t\t\t\t\t

Ideally, management training is applied on the job as situations arise that require it.\u00a0 One of the most effective ways to assess a manager’s comprehension is to use scenario-based questions that simulate real work experiences or actual work-related challenges that are likely to occur. Here is an example of a question stems that demonstrates how to upgrade a question from the recall to a higher-order thinking level using scenarios from\u00a0Malamed (2021b) –<\/span><\/p>

  • Before:<\/b>\u00a0What is the first concern of an emergency worker?<\/span><\/li>
  • After:<\/b>\u00a0An emergency technician (ET) arrived at the scene of an accident. Three people were injured. One person had scrapes and bruises, a second was moaning in pain and a third was quickly losing blood from a skull injury. Why did the ET triage the skull injury as the highest priority?<\/span><\/li><\/ul>

    Here is a sample vignette that requires learners to\u00a0 recall and synthesize information from Scully (2017).<\/p>

    A 28-year-old primigravid woman is at 11 weeks\u2019 gestation. Medical history is unremarkable. Family history is unremarkable except that both of her brothers have intellectual developmental disorder, her mother died of breast cancer at age 55, and her father is estranged. No family health records are available.<\/em><\/span><\/p>

    Which of the following studies is appropriate?<\/span><\/strong><\/p>

      1. \u00a0 Amniocentesis for \u03b1-fetoprotein<\/span><\/li>
      2. \u00a0 Blood test for fragile X carrier status<\/span><\/li>
      3. \u00a0 Blood test for phenylketonuria carrier status<\/span><\/li>
      4. \u00a0 Chorionic villus sampling for chromosome analysis<\/span><\/li>
      5. \u00a0 Chorionic villus sampling for Duchenne\u2019s muscular dystrophy<\/span><\/li><\/ol><\/li><\/ol>
        Here is a 3rd example (Assessing Higher-Order Outcomes Using Multiple-Choice Questions<\/i>, 2021)<\/span>.<\/div>

        \u201cBecause of rapidly rising national defense expenditures, it is anticipated that Country A will experience a price inflation unless measures are taken to restrict the growth of aggregate private demand. Specifically, the government is considering either (1) increasing personal income tax rates or (2) introducing a very tight monetary policy.\u201d If the government of Country A wishes to minimize the adverse effect of its anti-inflationary policies on economic growth, which one of the following policies should it use?<\/span><\/p>

        a. The tight money policy because it restricts consumption expenditures more than investment.<\/span>
        b. The tight money policy, since the tax increase would restrict consumption expenditures.<\/span>
        c. The personal income tax increase since it restricts consumption expenditures more than investment.<\/span>
        d. Either the tight money policy or the personal income tax rate increase since both depress investment equally.”<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

        \n\t\t\t\t
        \n\t\t\t\t\t

        Drawing conclusions from visuals like job aids and charts<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
        \n\t\t\t\t
        \n\t\t\t\t\t\t\t\t\t

        Create multiple choice questions that require the learner to first interpret this information before they answer the question.\u00a0 For example, present a graph or ask them what actions they would take based on the data or present the job aid and ask them to use it to make a judgement call based on available information.\u00a0\u00a0<\/p>

        Refer to this question stem from Malamed (2021b) below.<\/p>

        • Before:<\/b>\u00a0What is\/are the function(s) of a computer\u2019s projecting mechanism? (Select all that apply.)<\/span><\/li>
        • After:<\/b> Using the repair flowchart shown here, what should you check if the monitor stops working? <– use job aids to answer a question.<\/span><\/li><\/ul>
          Here is a 2nd example\u00a0(<\/span>Assessing Higher-Order Outcomes Using Multiple-Choice Questions<\/i>, 2021)<\/span>–\u00a0<\/span><\/div>

          Without other data, which conclusion can you make from reviewing Figure 17?<\/span><\/strong><\/p>

          a. The average American uses more drugs than citizens of any country except the United Kingdom<\/span>
          b. The average Mexican or Chilean consumes fewer drugs than citizens from other countries.<\/span>
          c. Americans are more likely than residents of other countries to use new drugs.<\/span>
          d. Japanese have regulations that make it very difficult to obtain new drugs<\/span><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

          \n\t\t\t\t
          \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
          \n\t\t\t\t
          \n\t\t\t\t\t

          Formulate questions with the help of Instructional Design frameworks <\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
          \n\t\t\t\t
          \n\t\t\t\t\t\t\t\t\t

          A third approach is to use Instructional Design models to formulate higher level multiple choice questions.  These techniques are frequently used in educational institutions to help aid in the development of critical thinking skills in children.  And, in cases where multiple choice questions are primarily knowledge based or need to be systematically applied to large populations, these techniques provide a plethora of ideas and insights.   Question stems based on Bloom’s Taxonomy verbs are designed to target learning experiences ranging from remembering to creating.  Webb’s Depth of Knowledge (DOK) is a framework that encourages learners to extend their understanding to new situations.  These guidelines may be helpful.  Paul’s Elements of Reasoning may also provide useful insights on how to develop questions to build a learner’s ability to apply logical and fact-based reasoning to new situations.<\/p>\n

          Here is an example of leveling up a multiple choice question from a recall question to one that requires comprehension using Bloom’s Taxonomy (Assessing Higher-Order Outcomes Using Multiple-Choice Questions, 2021).<\/p>\n

          \n<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

          \n\t\t\t\t
          \n\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\"\"\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
          \n\t\t\t\t
          \n\t\t\t\t\t\t\t\t\t

          With well written multiple choice, you can go beyond this by asking learners to interpret facts, evaluate situations, explain cause and effect, make inferences, and predict results.<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t

          \n\t\t\t\t
          \n\t\t\t\t\t

          Sources:<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t
          \n\t\t\t\t
          \n\t\t\t\t\t\t\t\t\t

          Assessing Higher-Order Outcomes using Multiple-Choice Questions<\/i>. (2021). Office of Teaching and Learning. https:\/\/otl.uoguelph.ca\/teaching-learning-resources\/assessment-student-learning\/assessing-higher-order-outcomes-using<\/p>

          Blended Learning Resources. (2019, August 29). Landing – Developing multiple choice questions that test higher order thinking<\/i>. http:\/\/blendedlearningresources.co.za\/landing-developing-multiple-choice-questions-that-test-higher-order-thinking\/#toggle-id-2<\/p>

          Brookhart, S. M. (2010). How to Assess Higher-Order Thinking Skills in Your Classroom<\/i> (1st ed.). ASCD.<\/p>

          Malamed, C. (2021a, November 6). Writing Multiple Choice Questions For Higher Order Thinking<\/i>. The eLearning Coach. https:\/\/theelearningcoach.com\/elearning_design\/higher-order-multiple-choice-questions\/<\/p>

          Malamed, C. (2021b, November 14). 10 Rules For Writing Multiple Choice Questions.<\/i> The eLearning Coach. https:\/\/theelearningcoach.com\/elearning_design\/rules-for-multiple-choice-questions\/<\/p>

          Scully, D. (2017) “Constructing Multiple-Choice Items to Measure Higher-Order Thinking,” Practical Assessment, Research, and Evaluation<\/em>: Vol. 22, Article 4. DOI: https:\/\/doi.org\/10.7275\/swgt-rj52
          Available at: https:\/\/scholarworks.umass.edu\/pare\/vol22\/iss1\/4<\/p>

          Weimer, M., PhD. (2019, November 25). Multiple-Choice Tests: Revisiting the Pros and Cons<\/i>. Faculty Focus | Higher Ed Teaching & Learning. https:\/\/www.facultyfocus.com\/articles\/educational-assessment\/multiple-choice-tests-pros-cons\/<\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t

          \n\t\t\t\t\t\t
          \n\t\t\t\t\t
          \n\t\t\t
          \n\t\t\t\t\t\t
          \n\t\t\t\t
          \n\t\t\t\t\t

          Author Information<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t
          \n\t\t\t\t\t\t
          \n\t\t\t\t\t
          \n\t\t\t
          \n\t\t\t\t\t\t
          \n\t\t\t\t
          \n\t\t\t\t\t\t\t
          [simple-author-box]<\/div>\n\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/div>\n\t\t\t\t\t<\/div>\n\t\t<\/section>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"

          Special Study – While multiple choice tests are typically used for superficial recall, they can be transformed into an effective tool for assessing the manager\u2019s comprehension and critical thinking skills in a real-world context. Read this article to learn more.<\/p>\n","protected":false},"author":1,"featured_media":123,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[55],"tags":[],"class_list":["post-1453","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-instructional-design"],"_links":{"self":[{"href":"https:\/\/premierjones.com\/felicia\/wp-json\/wp\/v2\/posts\/1453","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/premierjones.com\/felicia\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/premierjones.com\/felicia\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/premierjones.com\/felicia\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/premierjones.com\/felicia\/wp-json\/wp\/v2\/comments?post=1453"}],"version-history":[{"count":0,"href":"https:\/\/premierjones.com\/felicia\/wp-json\/wp\/v2\/posts\/1453\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/premierjones.com\/felicia\/wp-json\/wp\/v2\/media\/123"}],"wp:attachment":[{"href":"https:\/\/premierjones.com\/felicia\/wp-json\/wp\/v2\/media?parent=1453"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/premierjones.com\/felicia\/wp-json\/wp\/v2\/categories?post=1453"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/premierjones.com\/felicia\/wp-json\/wp\/v2\/tags?post=1453"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}