Assessment & Curriculum Design · Diagnostic Practice

Reasoning-Aware Practice

Designing online math practice that collects evidence about a student's reasoning, not only whether the final answer is right

A single right-or-wrong answer hides what a student understands. One wrong final answer can be four different students:

Wrong methodPicked the wrong model or rule from the start.

Sign slipRight method, one arithmetic or sign error.

Unaccepted formReasoned correctly, entered an equivalent the item rejected.

Concept gapNever grasped the underlying idea.

Same wrong answer, four possible next steps. These three prototypes help distinguish them, drawing on thirteen years of separating method from accuracy as an examiner and years of using adaptive practice in the classroom. They form one loop.

1 · Detect

Where did it break?

A multi-step item that pinpoints the first unsupported step instead of marking only the final answer.

2 · Repair

Fix the prerequisite

A repeated error triggers one short, targeted intercept, then returns the learner to a confirmation item.

3 · Verify transfer

Same idea, new form

A brief check that understanding holds across table, graph, equation, and context, not just one representation.

A note on scope.

Each prototype is a working proof of concept, built around a single skill so the design decisions are visible and testable.
Turning them into something real means a full misconception taxonomy, evidence from actual student responses, and consistent behavior across a Grades 6–12 catalog.
That is the design work I’d want to do inside a team. A few more directions are sketched at the end.

1 · Detect — where did the reasoning break?

In a multi-step problem, a wrong final answer is a poor diagnosis: it collapses concept selection, setup, execution, and interpretation into one bit of information. This prototype records the first unsupported step, giving feedback and teacher analytics more precise evidence about where the reasoning may have broken down. Work through it and try erring at different stages.

2 · Repair — fix the prerequisite, then confirm

When a learner repeats a structural error, another full problem at the same level rarely helps, and a written explanation is easy to skim past after frustration. This prototype watches for one specific misconception, pauses for a single targeted micro-question, then returns to a confirmation item with progress preserved. The error it repairs is the one traced across grade levels in From Arrays to Factoring.

3 · Verify transfer — same relationship, different representation

A student can score well on separate skills involving a table, a graph, an equation, and a context without recognizing they are the same relationship. This prototype is a brief transfer check: connect the representations, and spot the one that only looks like it belongs. Representation coherence is the throughline of One Learner, Three Systems.

What the teacher sees

Each prototype is designed backward from a teaching decision. Collecting this evidence only matters if it changes what a teacher does next:

From DetectNot just “12 of 30 got it wrong,” but which stage broke — model, setup, solve, or interpret — per learner and across the class. A room failing at setup needs a different lesson than one failing at interpretation.

From RepairWhether a repeated misconception was actually repaired: did the learner clear the confirmation item independently after the intercept, or is the error still live? The difference between “resolved” and “still stuck.”

From Verify transferWhether understanding is real or representation-bound: a learner fluent with equations but lost on the graph of the same relationship needs connection work, not more drill.

Across all threeEvidence a teacher can act on in a two-minute glance, paired with a learner-facing loop that repairs in the moment, so the evidence is used, not just recorded.

Further directions I’d develop

The three prototypes above are the ones most tied to item and feedback design, so they are the ones I built. Four more hypotheses I would develop and validate the same way, each scoped to answer a specific teaching question:

Best · current · stable signalsGive teachers three readings instead of one live score: the highest level a learner reached, their current state, and stability across recent independent attempts. A later slip stops erasing evidence of earlier mastery.

A "Review and Repair" queueEnd a session with two or three of the learner’s own mistakes, asking them to fix the first wrong step or reconstruct the expression, not simply redo the item. Distinguishes "repaired independently" from "still unresolved."

Finite, teacher-configurable practice contractsLet a teacher set the goal and stopping point (twelve focused minutes; eight scored questions; two independent challenge items) so an assignment has a clear purpose and endpoint while staying adaptive.

Optional "show your method" evidenceFor selected skills, capture the rule chosen or the first transformation, so analytics can separate "right formula, bad substitution" from "wrong model" without lengthening every item.

Each of these depends on the same core question this page is built around: what evidence would actually change a teaching decision, and how do we collect it without turning practice into a worksheet? That is the work I’d want to do from the inside.

What I’d validate before recommending any of it

None of this is a finished answer. Before proposing a feature, these are the questions I’d want evidence on:

Which existing capabilities already address part of the problem, so we build on them rather than around them?
Which secondary skills actually produce the most repeated-error loops or abandonment, and therefore deserve checkpoints first?
At what problem length does a checkpoint improve diagnosis enough to justify the extra interaction?
Can the system infer a single likely misconception reliably, or should it present several possible interpretations?
Do teachers actually decide differently when given peak, current, and longitudinal data separately?
Does each interaction stay keyboard-accessible and screen-reader-understandable, and does it preserve the mathematical construct?

Designing the study to answer these is as much the job as designing the feature.