learn-course-design-with-phoebe / Designer session 5 of 6
Learn Course Design with Phoebe · Designer track, session 5 of 6

Assessment and feedback

Most quiz questions in most training test whether somebody remembers a slide. That is a different ability from the one the objective named, and the gap between them is where the illusion of a successful course lives. Tonight: writing a question that tests the actual ability, building wrong answers out of the misconceptions people genuinely hold so that choosing one is diagnostic, and giving feedback that changes what happens on the next attempt. "Incorrect" is not feedback.

🟠 Hands-on Bring the ten real cases from a4 You will write questions tonight 45 min
0-3 · Welcome 3-18 · Testing the objective, and distractors that teach 18-40 · Build-along: write three questions 40-45 · Q&A
Part 0

Where we are

Session a2 decided what would prove the objective. Tonight you build the instrument, and the instrument is also your best teaching tool - a well-built question teaches more than the slide it replaced.

🎫 Clear all three quiz questions to stamp your passport Live = presented in the 45 minutes Self-study = the full depth, after class

★ What you walk out with tonight. Three questions that test your objective, distractors built from real misconceptions, and a feedback line for each wrong answer that says what to do differently.

LiveFormative and summative, and why one of them is optional2 min▶
  • Formative assessment steers. It happens during, it is low stakes, and its purpose is that you and the learner both find out what is not working while there is still time to do something.
  • Summative assessment certifies. It happens at the end and its purpose is a judgement.

Most workplace training has only the second, at the end, and often only as a completion tick. That is the wrong one to have if you can only have one. A course with only summative assessment cannot correct anything - by the time it tells you something, everybody has gone home.

If your session has one assessment moment, put it three-quarters of the way through, not at the end.

Part 1 · covers the question

Testing the ability, not the slide

6 min live
Objective: given a proposed schema change, classify it as safe or breaking Tests the slide "Which of the following are the six schema change types?" Closer "Is renaming a column a breaking change? Yes or no." Tests the ability "Here is a pull request. It adds order_total_v2, keeps order_total, and marks the old one deprecated. Safe or breaking? What would you tell the author?"
🔍 Click to zoom - only the third one puts them in the situation the objective described
LiveThree tests for a question4 min▶
  • Could somebody who can do the job get it wrong? If yes, the question has a trick in it and you are measuring your own cleverness. Remove the trick.
  • Could somebody who cannot do the job get it right? If yes - by elimination, by pattern, by remembering the emphasis you put on a word - the question is not measuring the ability.
  • Does it use the objective's condition? "Given a proposed schema change" means the question hands them a proposed schema change. If your objective has a condition and your question does not, they are testing different things.

The fastest way to fail the second test is to make the correct option the longest and most carefully qualified one. Everybody who has ever sat a badly written test knows to pick that one, and they do not have to know anything about your subject.

Part 2 · covers distractors and feedback

Wrong answers built out of real misconceptions

6 min live

A wrong option that nobody would ever choose is a wasted line. A wrong option that half the room chooses tells you exactly which misconception is in the room, which makes the question a diagnostic instrument as well as a test.

If they choose They believe So the feedback says "Breaking - it renames a column" any change to a published field breaks the old column is still there - nothing broke yet "Safe - both columns exist" additive is always safe, full stop safe now, breaking at the removal. Where is the date? "Safe - it is only a deprecation marker" deprecation is documentation, not a change a marker with no notice period is a plan, not a promise Every wrong option is a real thing an engineer at Daybreak has actually said. That is where distractors come from: not invented, collected.
🔍 Click to zoom - collect distractors from real errors, and the question starts diagnosing
LiveFeedback that changes the next attempt4 min▶

Three parts, and the third is the one that gets left out:

  • What was wrong. Specifically, not "incorrect". "You classified this as breaking."
  • Why. The misconception, named. "The old column is still present, so no consumer has broken yet."
  • What to do differently. The rule they can apply next time. "Ask what a consumer reading today would get. If the answer is unchanged, it is not breaking yet."

And the timing, which is a design decision rather than a preference: immediate for procedures, slightly delayed for judgement. When there is one right answer, tell them straight away. When it is a judgement call, let them commit and argue first - feedback that arrives before somebody has finished thinking replaces their thinking with yours.

The version that does nothing

"Incorrect. The correct answer is B." They now know they were wrong and have no idea why, so on the next case they will apply the same reasoning that just failed. This is what most quiz feedback in most systems looks like.

Build-along

Three questions, with distractors you collected

★ 22 min · everyone builds

Take a real case. One of the ten you collected in a4. Do not invent - a real case carries the ambiguity that makes the question worth asking.

Ask the ability. Use the objective's condition. Hand them the thing and ask them to do the verb.

Collect distractors. Not invent. Ask two colleagues to answer it and write down their wrong reasoning verbatim. That is your option list, and it is better than anything you would have written.

Write feedback per wrong answer. What was wrong, why, what to do differently. Three lines each. This is the part that is actually teaching.

Test it. Give it to somebody who can do the job and somebody who cannot. If the first gets it wrong or the second gets it right, the question is broken.

Write question one from a real case. Include the condition from your objective. It will be longer than a normal quiz question and that is correct - the situation is part of the test.

Go and collect wrong answers. Ask two colleagues, write their reasoning down verbatim. Real wrong reasoning is more plausible than anything you can invent, because it survived somebody's actual thinking.

Check your correct option is not the longest. If it is, everybody picks it without knowing anything. Rebalance the lengths.

Write three-part feedback for each wrong option. Name the misconception explicitly. "You may be assuming additive changes are always safe" does more work than any amount of re-explanation.

Decide the timing. Procedure, so immediate? Or judgement, so let them commit and argue first? Write it into the plan rather than deciding in the room.

Test both directions. One person who can do the job, one who cannot. Both results are informative and the second one is usually more surprising.

Now do two more. Three questions, well made, beat twelve improvised ones - and three is enough to tell you who is lost.

Your objective passes all four linter checks Observable verb, condition, criterion, short enough to test. From a1.
You wrote the proof before any content existed From a2. If content came first, the not-this list is probably empty too.
There are at least two worked examples, and the support fades From a3. Check you are not using the same rung for every activity.
The main activity's verb is the objective's verb From a4. The four-second check, and the most commonly failed one.
Every wrong answer has feedback naming the misconception What was wrong, why, what to do differently. "Incorrect" scores zero here.

If you take one thing from tonight. Collect your distractors instead of inventing them. Twenty minutes asking two colleagues gets you wrong answers that are genuinely tempting, and a question whose wrong answers are tempting is the only kind that tells you anything.

Homework

Before session a6

30 min
  1. Write three questions from real cases, with collected distractors and three-part feedback.
  2. Audit an existing quiz. How many of its questions test the slide rather than the ability? The fraction is usually uncomfortable.
  3. Find a "correct answer is longest" question in something you or somebody else wrote. They are everywhere.
  4. Test one question both directions - somebody able and somebody not - and note which way it failed.
  5. Bring everything to a6, which turns the design into materials for a room that will be shorter and more broken than the one you imagined.
Check yourself

Three questions

Clear all three and this session gets stamped in your passport.

1 · If your session can only have one assessment moment, where does it go?

Assessment at the end certifies but cannot correct - by the time it tells you something, everyone has gone home. Formative assessment steers, and steering requires there to be road left. A baseline is useful and it is not the one to have if you only get one.

2 · Where should the wrong options in a question come from?

Invented distractors are either obviously wrong, and therefore wasted, or accidentally ambiguous. Real wrong reasoning survived somebody's actual thinking, which makes it tempting - and a question whose wrong answers are tempting is the only kind that diagnoses anything.

3 · What is missing from "Incorrect. The correct answer is B."?

Three parts: what was wrong, why, and the rule to apply next time. Without the third, the person knows their answer failed and has no changed basis for the next one. Naming the misconception explicitly does more work than any amount of re-explanation.

Sources covered

What this session teaches, and where it came from

✓Formative against summativeOne steers and one certifies. A course with only the second cannot correct anything, and most workplace training has only the second
✓Testing the ability, not the slideThe same objective tested three ways, plus the three tests: no tricks, no free passes, and use the objective's condition
✓Distractors that diagnoseCollected from real wrong reasoning rather than invented, each mapped to the misconception it detects
✓Three-part feedbackWhat was wrong, why, what to do differently - with the timing rule: immediate for procedures, delayed for judgement
◐Certification, reliability, item analysisPsychometrics is a different discipline. Named where it starts, not taught

Every fact on this page, and its verification tier, is recorded in the course's source map. Daybreak's engineers and their misconceptions are written for teaching.

Designer a5 cheat sheet · pin this

Three-quarters, not the endAssessment that cannot be acted on is certification, not steering.
Use the objective's conditionIf the objective hands them a case, so does the question.
Both directionsCan an able person get it wrong? Can an unable person get it right?
Collect, do not inventAsk two colleagues and write down their wrong reasoning verbatim.
Not the longest optionEverybody picks the carefully qualified one without knowing anything.
Feedback in three partsWhat was wrong · why · what to do differently. "Incorrect" is not feedback.