# Item analysis: Building with the Claude API: knowledge check

- Learners: 80
- Items: 12
- Mean score: 7.25 of 12 (SD 2.34)
- Reliability (KR-20): 0.62
- Items flagged: 5

## All items

| Item | Key | Difficulty (p) | Discrimination (r) | D index | KR-20 if removed | Flags |
|---|---|---|---|---|---|---|
| Q01 | B | 0.97 | 0.16 | 0.09 | 0.62 | 5 |
| Q02 | B | 0.79 | 0.47 | 0.50 | 0.57 |  |
| Q03 | B | 0.68 | 0.47 | 0.68 | 0.56 |  |
| Q04 | B | 0.56 | 0.34 | 0.64 | 0.59 |  |
| Q05 | B | 0.38 | -0.16 | 0.00 | 0.68 | 3 |
| Q06 | B | 0.60 | 0.53 | 0.73 | 0.54 |  |
| Q07 | A | 0.47 | 0.32 | 0.68 | 0.59 |  |
| Q08 | C | 0.19 | -0.20 | -0.05 | 0.68 | 5 |
| Q09 | A | 0.42 | 0.22 | 0.64 | 0.61 |  |
| Q10 | B | 0.80 | 0.32 | 0.50 | 0.59 | 1 |
| Q11 | D | 0.62 | 0.48 | 0.82 | 0.56 |  |
| Q12 | B | 0.76 | 0.36 | 0.50 | 0.59 | 1 |

## Flagged items

### Q01: A response stops mid-sentence. Which stop_reason tells you the output hit the max_tokens limit you set?

| Option | Text | Picked | Top 27% | Bottom 27% | r with rest of test |
|---|---|---|---|---|---|
| A | end_turn | 1 (1%) | 0% | 5% | -0.11 |
| **B** (key) | max_tokens | 78 (98%) | 100% | 91% | 0.16 |
| C | stop_sequence | 1 (1%) | 0% | 5% | -0.11 |
| D | tool_use | 0 (0%) | 0% | 0% | n/a |

Flags:

- very easy: 98% answered correctly
- weak discrimination (r = 0.16)
- option A barely works as a distractor (1 picked it)
- option C barely works as a distractor (1 picked it)
- option D barely works as a distractor (0 picked it)

### Q05: Where does the main system prompt go in a Messages API request?

| Option | Text | Picked | Top 27% | Bottom 27% | r with rest of test |
|---|---|---|---|---|---|
| A | In the messages array, as a message with role "system" | 30 (38%) | 50% | 23% | 0.36 |
| **B** (key) | In the top-level system parameter | 30 (38%) | 41% | 41% | -0.16 |
| C | In the metadata field of the first user message | 8 (10%) | 5% | 14% | -0.14 |
| D | In a custom request header | 12 (15%) | 5% | 23% | -0.16 |

Flags:

- negative discrimination (r = -0.16): weaker learners did better than stronger ones
- option A attracts stronger learners (r = 0.36): possible miskey or a second defensible answer
- top scorers picked option A more often than the key

### Q08: Your app needs Claude's answer to match a JSON schema every time. What is the most reliable option?

| Option | Text | Picked | Top 27% | Bottom 27% | r with rest of test |
|---|---|---|---|---|---|
| A | Add "respond only in JSON" to the end of the prompt | 3 (4%) | 0% | 14% | -0.17 |
| B | Use structured outputs, for example output_config.format or messages.parse in the SDK | 54 (68%) | 77% | 41% | 0.35 |
| **C** (key) | Parse the reply with a regular expression and retry on failure | 15 (19%) | 23% | 27% | -0.20 |
| D | Give three JSON examples in the system prompt | 8 (10%) | 0% | 18% | -0.18 |

Flags:

- very hard: only 19% answered correctly
- negative discrimination (r = -0.20): weaker learners did better than stronger ones
- option A barely works as a distractor (3 picked it)
- option B attracts stronger learners (r = 0.35): possible miskey or a second defensible answer
- top scorers picked option B more often than the key

### Q10: Your app starts getting HTTP 429 responses. What do they mean, and what is the right response?

| Option | Text | Picked | Top 27% | Bottom 27% | r with rest of test |
|---|---|---|---|---|---|
| A | The API key is invalid, so rotate it | 6 (8%) | 0% | 23% | -0.19 |
| **B** (key) | You hit a rate limit, so back off and retry (the SDKs retry with exponential backoff) | 64 (80%) | 100% | 50% | 0.32 |
| C | The request body is too large, so split it | 8 (10%) | 0% | 18% | -0.18 |
| D | The model has been retired, so switch models | 2 (2%) | 0% | 9% | -0.14 |

Flags:

- option D barely works as a distractor (2 picked it)

### Q12: You submit 1,000 requests through the Message Batches API. How should you match each result to its request?

| Option | Text | Picked | Top 27% | Bottom 27% | r with rest of test |
|---|---|---|---|---|---|
| A | By position, since results come back in the order you sent them | 15 (19%) | 0% | 45% | -0.36 |
| **B** (key) | By the custom_id you gave each request | 61 (76%) | 95% | 45% | 0.36 |
| C | By the timestamp on each result | 4 (5%) | 5% | 9% | -0.05 |
| D | By input token count, since results are sorted by size | 0 (0%) | 0% | 0% | n/a |

Flags:

- option D barely works as a distractor (0 picked it)