To write multiple choice questions that measure knowledge, you need three things. A stem that asks a single question and makes sense without the options. One correct answer that no one can dispute. Wrong answers drawn from real mistakes, in the same form as the right one. Three options are enough. The rest is about removing the cues that let people answer without knowing.
What makes a multiple-choice question reliable?
A multiple-choice question has three parts: the stem, the correct answer and the wrong answers, or "distractors". The NBME, the US body that runs medical licensing exams, sorts item flaws into two families. It does so in the 4th edition of its item-writing guide (2016). Some flaws help test-wise candidates. Others make the question hard for reasons unrelated to the subject.
These flaws change who passes. In a study by Steven Downing of four medical school exams (2005), 36 to 65% of items had at least one flaw. They were 0 to 15 percentage points more difficult than sound items. Overall, 53% of students reached the passing score on the standard items, and 47% on the flawed ones.
The rules below also draw on the list of 31 guidelines validated in 2002 by Haladyna, Downing and Rodriguez. The NBME and Downing come from medical education, not workplace training. These rules cover questions with a single correct answer.
How to write the stem of a multiple-choice question (rules 1 to 4)
- The stem is enough to answer. Hide the options: a learner who knows should still be able to give the right answer (the "cover the options" rule in the NBME guide). So the central idea belongs in the stem.
- One question, one point from the course. Each question checks one objective of the module, not a detail from the course materials.
- No negatives, or bold them. Thomas Haladyna (2022) advises against "which is not" and "except". If you must use one, bold it, but don't rely on bolding alone. According to the NBME guide, some candidates miss the "except" even then.
- Repeated words go in the stem. If every option starts with the same words, move those words up into the stem.
Take a fictional company whose internal rule bans using the elevator during an alarm.
Before. Which of the following is not good practice? A. Go to the assembly point. B. Take the elevator. C. Close doors behind you. D. All of the above.
After. The fire alarm goes off while you're on the 3rd floor. How do you get down? A. By the nearest staircase. B. By the elevator, if it arrives right away. C. By the main staircase, after stopping at your desk.
In the first version, the stem asks no question until you read the options (rule 1). It is negative (rule 3), and "all of the above" makes no sense after a negative (rule 8). In the revised version, the stem stands on its own, and each wrong answer matches a real reflex: saving time, grabbing your things.
How to write good wrong answers (rules 5 to 8)
- Wrong answers drawn from real mistakes. Start from the confusions and misconceptions you hear in training. A wrong answer that no one picks is useless; a targeted mistake lets you give precise feedback.
- Same length, same form, same grammar. The NBME guide lists a correct answer that is longer, more specific or more complete as a cue. The same goes for the only option that fits the stem grammatically.
- No "always" or "never". The NBME also lists these absolute terms as cues: a test-wise candidate rules them out without knowing anything. Conversely, "sometimes" or "may" make almost anything defensible.
- No "all of the above" or "none of the above". Haladyna advises against both, because they give clues. The NBME guide calls "none of the above" problematic when the question involves judgment.
Same company, a question on expense claims.
Before. How long do you have to submit an expense claim? A. Within 30 days of the expense, with the receipt scanned into the expense tool. B. Never. C. Always on the same day. D. Whenever you like.
After. Under the company's procedure, how long do you have to submit an expense claim? A. 30 days after the expense. B. 30 days after the end of the month. C. 60 days after the expense.
In the first version, the correct answer stands out by its length (rule 6). The words "never" and "always" can be ruled out without knowing the answer (rule 7), and "whenever you like" tempts no one (rule 5). In the revised version, the options share the same form. The two wrong ones are real confusions: the end of the month, a former employer's deadline.
How many options, and where to put the right one? (rules 9 and 10)
- Three well-built options are enough. Michael Rodriguez's meta-analysis (2005) covers 80 years of research. It finds that three options let you ask more questions in the same time. Content coverage improves, with no harm to score quality. According to Haladyna, fourth and fifth options rarely discriminate or attract no one. To go from five to three, remove the two that no one picks, not two at random. If you have four credible ones, keep them.
- Vary the position of the right answer. Across the whole quiz, it shouldn't always land on B or C: the 2002 list makes this a rule. If order doesn't matter, a neutral order (chronological, numerical, alphabetical) avoids patterns.
With three options, a blind guess has a one-in-three chance of being right, but the extra questions make up for it.
Reviewing an AI-generated quiz: the checklist
Review each AI-generated question against these 8 points. They work just as well for a quiz you wrote yourself.
- Correct: the right answer is true according to your documents, not the AI's general knowledge.
- Only one: only one answer can be defended, even by an expert.
- Stands alone: the stem makes sense without the options (rule 1).
- Useful: the question checks a module objective, not a detail from the source document (rule 2).
- No form cues: the right answer is neither the longest nor the most specific (rule 6).
- No echo: no word from the stem appears only in the right answer.
- No automatic answers: no "always", "never", "all" or "none" (rules 7 and 8).
- Well spread: across the quiz, the right answer changes position (rule 10).
Complement's AI authoring tool generates end-of-chapter quizzes from your documents. Every question stays editable, by hand or with the AI editing button: the checklist is for that review. The dashboard then shows quiz success by chapter. A question that everyone fails, or everyone gets right, is the first one to review.
To try this review on a quiz generated from your own documents: 7-day free trial, no credit card required.