When the AI Gets It Wrong: Five Failure Modes We Plan For

We have read a lot of generated questions. They fail in patterns, and once you can name the patterns you can check for them in a fraction of the time. This is the list, with what each one looks like on the page.

1. The confident wrong key

What it looks like: a well-formed question with four sensible options and the wrong one marked correct. Nothing about it reads as suspicious. It is the most dangerous failure in the set, because the gentle version of review, scanning for things that look odd, will not catch it. Nothing looks odd.

Where it comes from: a calculation done in passing rather than worked through, or a fact recalled from training data rather than read from the chapter in front of it.

What catches it: answering the quiz yourself, with the key hidden, then comparing. Every disagreement is either an error or an ambiguity, and both need you. This takes five minutes for twenty questions and it is the single highest-value review habit there is.

2. Two defensible answers

What it looks like: the marked answer is right and so is another option, under a reading the question does not exclude. "Which material is best for a raincoat" with both plastic and nylon listed. "Which is the main cause" when the chapter gave three causes without ranking them.

Where it comes from: distractors generated to be plausible, pushed one step too far into being correct.

What catches it: reading each wrong option and asking whether a good student could argue for it. A student who picks a defensible wrong answer and is marked down learns the wrong lesson about their own reasoning, which is worse than learning nothing.

3. Difficulty drift

What it looks like: a set asked for at medium that is really a set of easy questions with longer sentences. Or the reverse in a smaller number of cases, where one question in twenty is two years beyond the class.

Where it comes from: difficulty is a judgement about a specific group of children, and the model has never met them. It writes to an average.

What catches it: reading the set as a whole rather than question by question. Ask where the hardest question is. In a medium set that is written properly, the last third should be noticeably harder than the first. If the set is flat, rewrite the end of it.

4. Recall instead of reasoning

What it looks like: twenty questions that all ask what something is called. Definitions, labels, names. Each one is fine. Together they test vocabulary and nothing else.

Where it comes from: definitional questions are the easiest to generate reliably from a text, so an unconstrained request drifts towards them.

What catches it: counting. How many questions could a student answer having memorised the glossary and understood nothing? If it is more than a third, the quiz needs application questions, which usually means writing two or three yourself. The mixed question types help here, since a sorting or ordering question is harder to answer from vocabulary alone.

5. Dead distractors

What it looks like: four options where two are obviously absurd. The question is nominally four-way and is really a coin toss.

Where it comes from: filling four slots when only two plausible wrong answers exist for this question.

What catches it: asking of each wrong option, which misunderstanding would lead a student to pick this? If no misunderstanding leads there, the option is doing nothing. A good distractor is a diagnosis: a student who picks it has a specific, nameable confusion, and that is information worth collecting.

The order to check in

If you have ten minutes for twenty questions, do this.

1. Answer them yourself with the key hidden. Compare. Investigate every disagreement. This finds failures 1 and 2.
2. Read the last five questions. Are they harder than the first five? This finds failure 3.
3. Count how many are pure recall. This finds failure 4.
4. If any time remains, read the wrong options on the questions that matter most. This finds failure 5.

That order is by cost of being wrong. A wrong answer key teaches a class something false. A dead distractor wastes a question. Both are worth fixing and only one is urgent.

What the system does and does not help with

It flags its own suspicious items, so a reviewer gets a short list to look at first rather than twenty items of equal weight. It restricts the shapes it can produce, so malformed output is dropped rather than shipped. It keeps every draft out of student view until someone decides. Those constraints are listed here.

What it cannot do is the judgement. Whether this wording suits this class, whether this is what was taught, whether a bright student could argue for option C. That is a person's job, and the reason the architecture is built around keeping a person in it.