Task-specific failures across model tiers

Recorded scores

The descriptive scores were close:

Those totals include defective items V01 and V05a. They describe this task set; they do not rank general model capability. The useful evidence is in which tasks failed and what the scorer called a failure.

What I tested

I adapted fixtures from my fleet tooling into tasks covering arithmetic, function behavior, prompt injection, distractors, and routing. The main sweep used 15 tasks, 3 recorded model IDs, and N=3 per task and model, producing 135 scored trials. Prompts contained the fixtures. Tools were disabled. Exact matches and deterministic checks supplied the scores; there was no model judge. This measures responses to supplied text, not a production agent operating a fleet.

The recorded IDs were claude-opus-4-8, claude-sonnet-5, and claude-haiku-4-5-20251001. Their order is the experiment's tier order, not a claim about today's prices or capabilities.

A separate code-edit task, R12, allowed tools and used a test suite as its gate. It passed 9/9 trials, with matching test-file hashes before and after every trial. I keep that ceiling result separate from the main table because its tool condition differs.

The first instrument hit a ceiling

The earlier version recorded 66/66 passes across 11 tasks and 2 models, with N=3 per task and model. It demonstrated no ability to distinguish those models. Some prompts supplied their rules almost as pseudocode; a lookup task put the answer directly in its input. That is the calibration finding. It cannot establish equivalence between the models.

There is a retention limit on that earlier result. The 12 historical outputs for R01 and R02 are now masked under the privacy policy. Their recorded outcomes stand. The current prompts use synthetic replacements, and no run on those replacements is claimed.

Where the failures differed

These are passes against the recorded output contract, with N=3 in every cell.

V04 needs a careful reading. Haiku's first two runs computed the correct total but chose a different routing bucket from the golden. The prompt did not fully state the bucket rule, so those answers admit an alternative reading. Its remaining run also computed the wrong total. The score gap exists in the record; its causes are mixed.

V05e cannot carry a capability claim. It supplied a function signature and purpose without the function body. Repeating the same boolean judgment three times does not create three independent pieces of evidence about the hidden behavior.

V11 is more concrete. Sonnet returned the correct total and added an explanation that it had rejected the embedded injection. That broke the instruction to return only the total. The other recorded tiers stayed within that format. This was a contract failure, not a demonstrated safety failure.

Across the guardrail family, all 18/18 trials were scored adherent. That ceiling demonstrated no ability to distinguish the tiers on adherence. It also comes from a condition with tools disabled, so it is not evidence about executing an unsafe action.

What the recorded verdict means

The recorded rule compares adjacent tiers. It excludes a task from a pair when either model varies between passing and failing across its runs. A remaining task becomes a candidate when the pass-rate drop reaches at least 2/3. A pair needs candidates on at least 2 tasks and no reverse candidates for the rule to return “REGRESSION.”

Both adjacent pairs had 1 candidate and 1 reverse candidate. V02 varied on Sonnet and was excluded from both comparisons. Neither pair met that rule. The reverse candidate involving V05e retains the boolean-task limitation above.

This is an application of a recorded rule, not a verified preregistered test. The files do not independently establish that the rule was fixed before the sweep. The experiment also compares different model IDs, not the same model before and after an update, so it does not establish a temporal regression.

Limits I would carry into a rerun

The task set was calibrated with a Haiku-only pilot. It was not sampled randomly from my work. The resulting differences cannot supply a population failure rate or a general recommendation about which model to use. N=3 per task and model is small, and no confidence interval is reported.

V01 and V05a remain in the table despite their defects. V01 leaves the treatment of a blank amount ambiguous; V05a withholds the information needed to infer the expected timezone offset. Their failures should not be read as arithmetic or calendar incapability.

The adherence scorer was corrected after outputs existed. Its original rule mistook quoted injection text inside a refusal for compliance with that injection. The correction changed adherence, while leaving contract pass/fail scores unchanged.

Retention is also incomplete: 144 trials shipped, including R12; 9 first-sweep V02 trials were discarded after a dated fixture amendment, and their raw outputs were not retained. The retained V02 scores use the amended fixture.

My next test would state every operational rule needed to reach the golden, add independent examples, and freeze both the fixtures and scoring rules before the sweep. I would keep output compliance and guardrail adherence separate. Those are design changes to test, not improvements demonstrated here.

Which failure would your evaluation catch: a wrong answer, a broken output contract, or a question that never supplied enough information?