# Claude item analysis

A small demo project that runs classical item analysis on a multiple-choice quiz, then asks Claude to read the flagged items like a subject-matter expert would and suggest what to do with each one.

The statistics tell you *that* a question misbehaves. Claude helps with *why*, because it can read the question text next to the numbers.

Everything in this repo is demo content. The quiz is a 12-question knowledge check on building with the Claude API, and the 80 "learners" are simulated by a script, with a few problems planted on purpose so there is something to find.

## What you'll learn

- What item analysis is and which numbers matter
- How to compute difficulty, discrimination and distractor statistics in plain Python
- How to send one flagged item to Claude with the Python SDK and get a structured review back (cause, action, revised item)
- How to handle refusals, cut-off output and API errors without crashing the run

## Quick start

You need Python 3.10 or newer.

```bash
pip install -r requirements.txt

# Stats only. No API key needed.
python analyze.py --no-claude

# Stats plus Claude's review of each flagged item.
export ANTHROPIC_API_KEY=your-key-here
python analyze.py
```

Both commands write `report.md`. A stats-only report from the sample data is in `sample-output/`.

To run the tests:

```bash
pip install pytest
pytest
```

## Project layout

```
analyze.py              command-line entry point
itemstats.py            the statistics, plain Python, no API calls
claude_review.py        the Claude step: prompt, output schema, error handling
report.py               builds the Markdown report
data/quiz.json          the sample quiz (stems, options, answer key)
data/responses.csv      simulated answers from 80 learners
scripts/make_sample_data.py   how the fake responses were generated
tests/                  checks for the statistics
```

## Part 1: item analysis in five minutes

Item analysis looks at how each question performed across everyone who took the test. Three numbers do most of the work.

**Difficulty (p)** is the share of learners who got the item right. An item at 0.97 tells you almost nothing, because nearly everyone gets it. An item at 0.15 might be testing something nobody was taught, or it might be broken. Most well-built items land between 0.30 and 0.90.

**Discrimination (r)** asks whether the item separates people who know the material from people who don't. Here it is the corrected point-biserial: the correlation between getting this item right and the learner's score on the *rest* of the test. Leaving the item out of the total keeps it from correlating with itself.

- 0.30 and up: good
- 0.20 to 0.30: acceptable
- below 0.20: weak
- negative: weaker learners did better than stronger ones, which almost always means something is wrong with the item

**Distractor analysis** looks at the wrong options. For each option the script reports how many people picked it, how often it was picked by the top 27% and bottom 27% of scorers, and its own point-biserial. A healthy distractor pulls in the lower group and has a negative point-biserial. Two patterns are red flags:

- A wrong option that the top group prefers, or that has a positive point-biserial. That usually means the key is wrong, or there are two defensible answers.
- A wrong option almost nobody picks. It isn't doing any work, so the item is easier than it looks.

The script also reports **KR-20**, the reliability of the whole test, and what KR-20 would be if each item were removed. If removing an item makes the test *more* reliable, that item is hurting more than helping.

All of this lives in `itemstats.py`. It uses only the standard library, so you can read it top to bottom. The thresholds are constants at the top of the file, and they are rules of thumb, not laws.

## Part 2: what the sample data shows

Run `python analyze.py --no-claude` and five items get flagged:

| Item | What the numbers say |
|---|---|
| Q01 | 98% correct and two of the three distractors drew one learner each. Too easy to tell anyone apart. |
| Q05 | Negative discrimination. Strong learners split between the key (B) and option A. |
| Q08 | 19% correct, negative discrimination, and the top group overwhelmingly picked B instead of the key (C). |
| Q10 | Option D drew only 2 of 80 learners. |
| Q12 | Option D drew nobody. |

The numbers can't tell you what to do about Q05 or Q08. Is Q08 miskeyed, or is it a hard item that strong learners overthink? You have to read it. That is where Claude comes in.

(Spoiler, because the data is simulated and we know the truth: Q08 is miskeyed in `quiz.json`, and Q05 is ambiguous for people who know that the API also accepts `system` role messages later in a conversation.)

## Part 3: adding Claude

`claude_review.py` sends each flagged item to Claude, one request per item. The request has three parts.

**A system prompt that teaches the job.** It explains what each statistic means and what the thresholds are, then asks Claude to do what a reviewer does: read the item as a subject-matter expert, decide whether the key and options are really correct, name the likely cause, and recommend one action. It also says not to invent numbers.

**The item and its statistics as JSON.** `build_item_payload()` packs the stem, options, key, objective, all the statistics and the script's flags into one JSON string. Giving Claude the raw numbers, not just the flags, lets it weigh the evidence itself.

**A schema for the answer.** Instead of asking for JSON in the prompt and hoping, the code uses structured outputs. The `ItemReview` Pydantic model defines the fields, and `client.messages.parse()` turns it into a JSON schema, sends it with the request, and hands back a validated object:

```python
class ItemReview(BaseModel):
    item_id: str
    likely_cause: Literal["miskey", "ambiguous_or_two_correct_answers", "too_easy",
                          "too_hard", "weak_distractors", "confusing_wording",
                          "no_real_problem"]
    diagnosis: str
    action: Literal["keep", "rekey", "revise", "retire"]
    revised_item: RevisedItem | None = None
    what_changed: str

response = client.messages.parse(
    model="claude-opus-5-5",
    max_tokens=16000,
    system=SYSTEM_PROMPT,
    output_config={"effort": "medium"},
    output_format=ItemReview,
    messages=[{"role": "user", "content": "Review this item...\n\n" + payload}],
)
review = response.parsed_output
```

Using `Literal` for `likely_cause` and `action` means Claude has to pick from a fixed list, so the report can sort and count them. `effort` controls how much thinking Claude does before it answers. Medium is plenty for one quiz item.

### Handling what can go wrong

A batch job should finish even when one request doesn't. The code checks two stop reasons before it trusts the output:

- `refusal`: Claude declined the request. The item is skipped with a message.
- `max_tokens`: the answer was cut off, so the JSON may be incomplete. Skipped too.

It also catches the SDK's typed errors, most specific first. The SDK already retries rate limits (429), overloads and server errors with exponential backoff, so a `RateLimitError` that reaches your code means the retries ran out, and the run stops cleanly instead of hammering the API. Other `APIStatusError`s, like a 400 for a bad request, skip just that item, since retrying the same request won't help.

The model's `item_id` field is overwritten with the id the code actually asked about. A small thing, but it means a typo in the model's output can never attach a review to the wrong question.

## Part 4: using your own quiz

Make a quiz file shaped like `data/quiz.json` (an `id`, `stem`, `options` and `key` for each item; `objective` is optional but helps Claude), and a CSV with a `learner_id` column plus one column per item id holding the letter each learner chose. Then:

```bash
python analyze.py --quiz my_quiz.json --responses my_responses.csv --out my_report.md
```

A few cautions:

- Classical statistics get noisy with small groups. Under about 30 learners, treat the flags as prompts to look, not verdicts.
- Claude's review is a second opinion for a human reviewer, not a replacement for one. Check every suggested rekey against your source material before you change a live test.
- Don't send learner names or IDs to the API. Only item text and aggregate statistics leave your machine in this project.

## Ideas for extending it

- Send all flagged items in one Message Batches request to cut cost in half on large item banks.
- Cache the system prompt with prompt caching once it grows past the minimum cacheable length.
- Add a check that runs the revised item past Claude a second time, as a fresh reviewer, before you accept it.

## License

MIT. See `LICENSE`.