Claude Item Analysis

SEC. 01The Problem

Every assessment has a few questions that quietly don't work. Some are so easy they measure nothing. Some have a distractor nobody picks. And some are worse: the answer key is wrong, or there are two defensible answers, and the people who know the material best are the ones getting marked wrong.

Classical item analysis is good at finding these. It is not good at explaining them. A negative discrimination value tells you something is off, but you still have to read the question to find out whether it is a miskey, an ambiguous stem, or a hard item that strong learners overthink. That reading step is where this project brings in Claude.

SEC. 02How It Works

The pipeline runs in four steps:

1. Load the quiz (stems, options, answer key) and a CSV of learner responses
2. Compute difficulty, corrected point-biserial discrimination, upper and lower group stats, distractor stats and KR-20 reliability
3. Flag items that cross common thresholds, in plain language
4. Send each flagged item, with its text and statistics, to Claude for a structured review

The statistics are plain Python with no dependencies, so the first three steps run without an API key. The Claude step is optional and lives in its own file.

SEC. 03The Demo Data

The sample quiz is a 12-question knowledge check on building with the Claude API: stop reasons, tool use, prompt caching, structured outputs, rate limits and batches. The 80 learners are simulated with a standard three-parameter logistic model, and four problems are planted on purpose: one item that is far too easy, one that is ambiguous, one that is miskeyed, and one with a dead distractor.

Because the data is simulated, the right answers are known in advance. That makes it easy to check whether the statistics and Claude's review actually catch what was planted.

SEC. 04Adding Claude

Each flagged item goes to Claude in its own request. The system prompt explains what each statistic means and asks Claude to do what a human reviewer would: read the item as a subject-matter expert, decide whether the key and options are really correct, name the likely cause, and recommend one action.

Instead of asking for JSON and hoping, the project uses structured outputs. A Pydantic model defines the review, and the SDK turns it into a schema that Claude's answer has to match:

# claude_review.py
class ItemReview(BaseModel):
    item_id: str
    likely_cause: Literal["miskey", "ambiguous_or_two_correct_answers",
                          "too_easy", "too_hard", "weak_distractors", ...]
    diagnosis: str
    action: Literal["keep", "rekey", "revise", "retire"]
    revised_item: RevisedItem | None = None
    what_changed: str

response = client.messages.parse(
    model="claude-opus-5-5",
    max_tokens=16000,
    system=SYSTEM_PROMPT,
    output_config={"effort": "medium"},
    output_format=ItemReview,
    messages=[{"role": "user", "content": payload}],
)
review = response.parsed_output

Fixed lists for the cause and the action mean the report can sort and count reviews, and every review has the same shape no matter what Claude decides.

SEC. 05When Things Go Wrong

A review run over a whole item bank should finish even when one request doesn't. The project handles the cases a real run hits:

• Refusals and cut-off output. It checks stop_reason for refusal and max_tokens before trusting the parsed result, and skips that item with a message.
• Rate limits. The SDK already retries 429s and server errors with backoff, so a rate-limit error that reaches the code means the retries ran out, and the run stops cleanly.
• Bad requests. Other API errors skip just that item, since sending the same request again won't help.
• Mixed-up ids. Each review is attached to the item the code asked about, not the id Claude echoes back.

SEC. 06Try It

pip install -r requirements.txt

# stats only, no API key needed
python analyze.py --no-claude

# stats plus Claude's review of each flagged item
export ANTHROPIC_API_KEY=your-key-here
python analyze.py

Both commands write a Markdown report. To use your own quiz, point --quiz and --responses at your files. The README walks through the file formats, what each statistic means, and the cautions that matter, like small sample sizes and keeping learner names out of anything sent to the API.

SEC. 07The Code

The whole project is on this site. Download it as a zip, or read any file below.

Released under the MIT license. All quiz content and learner data in the project are made up for the demo.