I asked for shorter worker replies. Solo still cost less.
I added a compact return format to my orchestration benchmark. The solo agent still cost less and finished faster in both completed runs.
That was the question behind the change: could shorter worker replies close the delegation cost gap without losing answer quality? The cost gap remains. Later quality judgments varied by comparison and run.
The comparison
I ran the same scripted audit across three arms. Solo had delegation tools disabled. Standard orchestration instructed the main agent to delegate repository reading and searching. The tight-return arm added a compact file:line format and a worker-return token cap to that instruction.
Each session mapped configuration, order paths, safety checks, and documentation gaps, then produced an audit summary and a recall answer. Runs were interleaved across arms, with the same main-model alias and task sequence. The instruction was to return findings without modifying files.
Two sessions per arm completed every scripted turn. Each contained six scripted turns. A weekly usage limit interrupted the next run, so every partial and error record from that run is excluded from the comparison.
The receipts
Mean per completed session, N=2 per arm:
| Arm | Usage valuation, USD | Invocation wall time, seconds |
|---|---|---|
| Solo | 8.1771 | 449.55 |
| Standard orchestration | 19.5365 | 1286.35 |
| Tight-return orchestration | 17.3466 | 859.05 |
USD values are truncated to four decimal places. The evidence retains full precision. They represent the CLI's usage valuation under a subscription. Wall time sums the measured invocations and excludes filesystem snapshots, gaps between sessions, and later judging.
The tight-return arm had lower mean cost and wall time than standard orchestration. It remained more expensive and slower than solo. Neither delegation arm reached a lower mean cumulative cost than solo at any measured turn.
The session ranges matter:
| Arm | Usage valuation range, USD | Invocation wall range, seconds |
|---|---|---|
| Solo | 6.8717 to 9.4826 | 379.5 to 519.6 |
| Standard orchestration | 19.1396 to 19.9333 | 1207.7 to 1365.0 |
| Tight-return orchestration | 14.9181 to 19.7752 | 840.0 to 878.1 |
Later quality judgments
Codex later scored the saved Claude audit summaries and recall answers for completeness, correctness, and grounding, with recorded source-citation checks. This was a post hoc change from the planned Claude assessor. The original benchmark sessions and their cost accounting stayed unchanged.
The comparisons used anonymous X/Y labels and the original scoring rubric. Worker references remained in the saved answers, so the judge could infer the approach from the text. Label masking did not guarantee successful blinding.
| Comparison | Run 1: Codex preferred | Run 2: Codex preferred |
|---|---|---|
| Solo vs standard orchestration | Standard orchestration | Solo |
| Solo vs tight-return orchestration | Solo | Solo |
| Standard vs tight-return orchestration | Standard orchestration | Tight-return orchestration |
Solo was preferred to tight returns in both runs. The other comparisons split. Each captured session appears in two comparisons; the results retain both sets of scores and their disagreements. Those repeated judgments do not add independent benchmark sessions.
One tight-return recall result contains only a completion acknowledgment referring to an answer absent from the captured text. It was judged as stored. These scores describe captured final responses, with a known capture gap; they cannot establish that a cost change preserved the quality of all work performed.
Guardrails and measurement limits
The factual-error screen flagged 1 of 2 solo sessions, 0 of 2 standard-orchestration sessions, and 2 of 2 tight-return sessions. It triggers on a correctness score below 7 or selected keywords in the judge's notes, including matches inside negated wording. A flag is not a confirmed-error count, and a clear screen does not prove an error-free answer.
The audit-summary check found no qualifying fenced blocks of code. That is a narrow observation. It does not tell me whether internal workers honored the return-length instruction, and the unrestricted arms provide only a baseline for this check.
The filesystem detector also needed interpretation. Known background activity changed the same operational files while the benchmark ran. After excluding those documented paths, one other file changed during standard orchestration's first run. The saved fingerprints establish the change but cannot identify its writer.
I keep that residual observation in the record. A detector flag alone does not establish an agent-caused write, and a clear net count cannot prove that nothing changed outside the monitored repository.
What this changes
My earlier benchmark had one completed session per arm. Solo was cheaper, faster, and favored by the blind quality judgment. That result still stands for that experiment.
The newer run uses a different Opus identifier. That model change prevents a controlled comparison with the earlier experiment. The later Codex judge is another change, so its scores cannot be compared directly with the historical Claude judgments.
This is a descriptive result from one repository and two completed sessions per arm, with no confidence interval. The measured cost and timing leave a useful next question: which part of the delegation overhead can be reduced while keeping the answer useful?
Would you test worker scope, return size, or orchestration overhead next?