One of our most dedicated users, a nursing student, sent us two lines of feedback:
the choices for multiple choice questions need to be harder. The correct answer is too obvious because the other choices make no sense
My first reaction was a bit defensive. The questions come from her own lecture slides, and they're written by a good model with a long list of rules about variety and accuracy. Surely they're fine?
So I pulled every multiple-choice question The Study Mill had ever written. 1,761 of them, from subjects ranging from nursing to law. Then I asked one boring question: where is the right answer?
Option B. Almost three times out of four.
If you'd simply picked B on every question, you'd have scored 72% without reading a word.
The second tell was length
Position was the obvious one. The sneakier one was length.
The right answer was the longest option 67% of the time (on her questions, 81%). On a four-option question, a fair quiz would land there about 25% of the time. On average the correct option was 1.4 times longer than the wrong ones.
That makes sense once you think about how a writer, human or machine, builds a question. The right answer has to be true, so it collects qualifiers: "usually", "in most patients", "when combined with". The wrong answers only have to be wrong, so they get written quickly and come out short. Here's one it wrote, lightly trimmed:
What does a nursing diagnosis identify? A. The specific medications the patient needs B. How the disease affects the patient's daily life and where nursing care can help C. The cellular cause of the condition D. The patient's expected recovery timeline
You don't need to know any nursing to guess B. It's the long, careful, reasonable-sounding one.
Why this matters more than it sounds
A quiz that you can beat by pattern-matching isn't testing what you know. Worse, it feels like it is. You get 9 out of 10, you feel ready, and then the real exam (written by a person who has seen this trick before) catches you out.
Practice questions are one of the best study tools there are. In learning research, testing yourself consistently beats rereading. But only if the questions make you actually retrieve the answer. If the answer is sitting there being the longest, you're not retrieving. You're spotting.
What we changed
Three fixes, in order of how much they mattered:
1. We shuffle the options ourselves. We stopped trusting the model's order. Every question's options now get sorted by a fingerprint of the question and the option text, which spreads the answer evenly across A to D and keeps it in the same place every time you reload. This applied to all 1,761 existing questions straight away.
2. We tightened the rules for wrong answers. Every wrong option now has to be a real mistake a half-prepared student would make: a commonly confused concept, a reversed cause and effect, a step in the wrong order. No filler, no "all of the above", and all options roughly the same length.
3. We check the length, and fix it if the rules didn't work. This was the surprise. Even with explicit instructions, the model left the right answer as the longest option about 60% of the time. So now, after a set is written, any question where the answer still gives itself away by length gets its wrong options rewritten to match. The right answer itself is never touched.
After those changes, on fresh sets: the right answer is the longest option 19% of the time (fair is 25%), it's no longer any longer than the wrong options on average, and it lands in every position. That's on a small test sample of new questions, so we'll keep watching it.
We also added a button she asked for in the same message: Swap for new ones, which replaces a quiz with a brand new set instead of piling more lookalike questions on top.
How to check any AI quiz in two minutes
This isn't a Study Mill problem. It's how language models write multiple choice, so it shows up in any quiz an AI writes for you, including ones you make in ChatGPT. Before you trust a set of AI practice questions, run this:
- Take 10 questions and answer them without reading the question. Just pick the longest option each time.
- Do it again picking B every time.
- If either strategy beats about 40%, the quiz has a tell. Your score on it means less than you think.
If you're making questions yourself with a chatbot, add this to your prompt: "Make every option the same length and level of detail, make each wrong answer a common misconception, and randomise which letter is correct." It helps, though as we found, it won't fully fix it.
The bit I'm grateful for
We'd never have looked if a student hadn't taken the time to tell us. Two lines of feedback led to a fix for every student on the app. If something in The Study Mill feels off to you, tell us. A real person reads it, and sometimes it turns into this.
Want to see the new questions on your own lecture? Upload one and try to beat the quiz without reading it. You shouldn't be able to anymore.
Method, for the curious: every multiple-choice question stored in The Study Mill on 11 October 2026 (1,761 questions; automated test accounts excluded). "Longest" means the correct option was strictly longer than every other option.
