Accuracy is Intelligence
ResultsWhyWhere they failCost and timeTasks
// SteelEnv · computer-use track · five-task study · November 2026

Five hard steel tasks in FreeCAD, six model settings

We ran GPT-6 Astra, Claude Fable 5.1 and Claude Opus 5.5 at max effort on the five rebuild tasks where the earlier models did worst, and compared them with the earlier Opus 5.5 high, GPT-6.1 Sol and Gemini 4 Argon runs: scores, where each one fails, why, and what it costs.
Deccan AI · 5 tasks · 6 model settings · 90 runs, 83 finished and graded · November 2026

On five rebuild tasks picked because the earlier models did badly on them, Claude Opus 5.5 at max effort solved 7 of 8 of its finished runs. Gemini 4 Argon solved 8 of 15, Claude Fable 5.1 6 of 15, GPT-6 Astra 5 of 15, the same Opus 5.5 at high effort 4 of 15 and GPT-6.1 Sol none. Max effort is also the slowest and dearest by far: about 11,000 tokens of thinking per step, 13 hours and $140 per finished run.

7 of 8
finished Opus 5.5 max runs solved, the best result
4 of 15
Opus 5.5 high runs solved on the same tasks
5 of 15
GPT-6 Astra (high) runs solved
$4,691
spent on model calls for all 90 runs
1 · Which setting is best2 · Why: what the reasoning and screens show3 · Where each model fails4 · Cost and time5 · The five tasks6 · How we ran itAppendix · every run

1 · Which setting is best

Ranked by the share of finished runs that solve the task. Opus max leads on every task it finished; its stopped passes were halted by us, not by failure. Astra's one-run lead over Opus high comes from the stair stringer, which it solved 3 times in about 20 minutes each.

Model and effort
Runs solved
Average score
Cost / run
Time / run
Claude Opus 5.5 maxstudy
7 of 8
0.98
$120
13.2 h
Gemini 4 Argon highearlier
8 of 15
0.74
$7
2.5 h
Claude Fable 5.1 highstudy
6 of 15
0.72
$36
1.4 h
GPT-6 Astra highstudy
5 of 15
0.61
$48
1.2 h
Claude Opus 5.5 highearlier
4 of 15
0.59
$18
1.1 h
GPT-6.1 Sol highearlier
0 of 15
0.53
$6
1.1 h

Finished runs only. A run is solved at score 1.0: every part within 1/16 in, every hole right, no clash. Cost and time are medians over finished runs. Opus max finished 8 runs; 7 more were stopped before they ended (section 6).

The pipe-support frame (r9) was re-graded on 3 Nov. The grader's overlap test failed on beams that touch the existing angles face to face: after its shape check had meshed the beam, it counted two clip angles as lying entirely inside it (2 × 218.7 = 437.3 cm³). That false clash was in 11 of the 18 runs. With the test run on clean copies of the solids, 8 scores go up and 4 runs that had built every part right now score 1.0: Argon · high pass 1, Fable 5.1 · high pass 2, Fable 5.1 · high pass 3, Opus 5.5 · high pass 3. Nothing else changed, and the unchanged grader run the same way reproduces every official verdict. All scores on this page include the re-grade.

Every run at a glance

One box per run, three runs per model per task. Hover or tap a box for its details.

Opus maxmax
Argonhigh
Fablehigh
Astrahigh
Opus highhigh
Solhigh
Skewed rooftop pipe-support framev4r07 · 10 parts
✓
1/1
.00.00✓
1/3
✓.32.72
1/3
.00.00✓
1/3
.45.36✓
1/3
.58.44.44
0/3
Double-angle X-bracing in a factory wallv4r02 · 6 parts
✓
1/1
.60.60✓
1/3
✓✓.00
2/3
.30.60✓
1/3
.75✓.60
1/3
.60.15.30
0/3
Stair stringer with its tread anglesv5r04 · 14 parts
✓
1/1
✓✓✓
3/3
✓.06.78
1/3
✓✓✓
3/3
.78✓.06
1/3
.66.78.78
0/3
Pipe-support frame on a pipe rackr9 · 5 parts
✓✓✓
3/3
✓✓.54
2/3
.72✓✓
2/3
.72.00.72
0/3
.54.04✓
1/3
.30.72.45
0/3
Stair landing post framesv5r09 · 12 parts
✓.82
1/2
.60.83✓
1/3
.82.53.82
0/3
.75.60.45
0/3
.60.56.06
0/3
.75.53.45
0/3
✓ solved (1.0) not solved: the fill is the score (0.9 × share of parts right) stopped before it finished

Opus 5.5: high effort against max effort

Same model, same tasks, same prompts and grader; only the effort setting differs. At max effort Opus solved every task it finished, including the stair landing frames, which it never solved at high effort.

Skewed rooftop pipe-support framev4r07 · 10 parts
high
1/3 solved
max
1/1 solved +2 stopped
Double-angle X-bracing in a factory wallv4r02 · 6 parts
high
1/3 solved
max
1/1 solved +2 stopped
Stair stringer with its tread anglesv5r04 · 14 parts
high
1/3 solved
max
1/1 solved +2 stopped
Pipe-support frame on a pipe rackr9 · 5 parts
high
1/3 solved
max
3/3 solved
Stair landing post framesv5r09 · 12 parts
high
0/3 solved
max
1/2 solved +1 stopped
00.250.50.751.0 solved

Bar = average score of the finished runs; dots = each run (filled = solved). "Stopped" passes were halted by us before they finished: six once another pass had solved the task, one after the API usage limit (section 6).

2 · Why: what the reasoning and screens show

Astra is fast and literal: it starts building within minutes, copies parts that look right and rarely checks, so it wins where quick, exact copying is enough. Opus at high effort reads carefully, but its thinking often spots a problem and moves on. At max effort it measures everything in the model and checks its work before finishing, which is why it solved 7 of 8, at 9 to 26 hours a run.

GPT-6 Astra · highOpus 5.5 · highOpus 5.5 · max
Runs solved5 of 154 of 157 of 8
First part builtafter 5 to 18 minafter 10 to 80 minafter 3 to 14 h
Reading the drawingsfast and literal; copies parts that look alikecareful, but misreads small tables and uses handbook sizesthorough, then measures the model to settle every doubt
Checks before "done"rarelyrarely; its thinking notices problems and moves onalways: measures, reopens its export, zooms in
Final messageone line: "done"lists what it did not checkstates what it checked
Output per stepabout 210 tokensabout 500 tokensabout 11,000 tokens

Skewed frame: Astra 1 of 3, Opus high 1 of 3, Opus max 1 of 1

Parts already in the model look like the missing girts but sit at slightly different spots. Astra copied one of them in two runs instead of reading the drawing, so all 10 parts slid 33 mm (score 0). Opus high twice took the lower girt's height from a nearby plate instead of the drawing; Opus max checked both girt heights against the drawing before saving.

Opus high, pass 1: it checks the lower girt's height on a knife plate (status bar: z = 567,080.4), not the drawing's 567,055. The girt ends up 12.7 mm high.
Opus high, pass 1: it checks the lower girt's height on a knife plate (status bar: z = 567,080.4), not the drawing's 567,055. The girt ends up 12.7 mm high.
Opus max: before saving it hovers its own girt top and reads z = 569,341.0000, exactly the drawing's elevation (and 567,055.0 for the lower girt).
Opus max: before saving it hovers its own girt top and reads z = 569,341.0000, exactly the drawing's elevation (and 567,055.0 for the lower girt).

X-bracing: Astra 1 of 3, Opus high 1 of 3, Opus max 1 of 1

All three read the drawings right. The score came down to a FreeCAD quirk: one placement box shows the number you typed but stores it rounded, which pushes a brace 0.05 mm into the plate it bolts to, just over the clash limit. Astra caught it once by reading the stored value back, Opus high twice called the rounded value “a match”, and Opus max typed its numbers where FreeCAD keeps them exactly.

Opus high, pass 1: it typed 97,326.453; the panel shows the stored 97.326500 m and Opus calls it a match. That 0.05 mm fails the clash check.
Opus high, pass 1: it typed 97,326.453; the panel shows the stored 97.326500 m and Opus calls it a match. That 0.05 mm fails the clash check.
Astra, pass 3: the panel shows y = 97,288.400000, not the 97,288.35 it typed. It retypes the value, and this one check turns 0.60 into 1.00.
Astra, pass 3: the panel shows y = 97,288.400000, not the 97,288.35 it typed. It retypes the value, and this one check turns 0.60 into 1.00.

Stair stringer: Astra 3 of 3, Opus high 1 of 3, Opus max 1 of 1

Every run copied the stringer on the other side of the stair as a mirror image, so the score came down to two details: the mirror line within 0.8 mm, and exactly the 14 parts on the parts list. Astra measured the tread itself and checked each part's size, about 20 minutes a run; Opus high once measured from the wrong edge (below) and once copied a wrong plate; Opus max worked the line out from the drawing first, then measured to confirm.

Opus high, pass 3: it takes the end of the tread from the steel edge it is hovering (status bar: x = 22631.40). The tread really starts 1.6 mm further on, so all 13 copied parts land 1.6 mm off.
Opus high, pass 3: it takes the end of the tread from the steel edge it is hovering (status bar: x = 22631.40). The tread really starts 1.6 mm further on, so all 13 copied parts land 1.6 mm off.
Astra, pass 2: it hides the steel and measures the tread's own corner, x = 22632.99. Its copies land in the right place.
Astra, pass 2: it hides the steel and measures the tread's own corner, x = 22632.99. Its copies land in the right place.

Pipe-support frame: Astra 0 of 3, Opus high 1 of 3, Opus max 3 of 3

Astra lost the same small vent hole in all three runs: it edited two hole cutters at once, FreeCAD wrote one position into both, and Astra never looked at the finished post. Opus high misread the drawing twice (it skipped the vent holes once and set the posts 229 mm inboard once); Opus max measured every face and slot, then zoomed in on its own posts before saving.

Astra, pass 1: two hole cutters are selected, but the panel shows only one position. The height it types next goes into both, and one vent hole disappears.
Astra, pass 1: two hole cutters are selected, but the panel shows only one position. The height it types next goes into both, and one vent hole disappears.
Opus max, pass 3: before saving it zooms in on its new post (four vent holes) and checks the beam holes sit over the existing slots.
Opus max, pass 3: before saving it zooms in on its new post (four vent holes) and checks the beam holes sit over the existing slots.

Re-graded on 3 Nov (section 1): Opus high pass 3 and Fable passes 2 and 3 built every part right and now score 1.0.

Stair landing: Astra 0 of 3, Opus high 0 of 3, Opus max 1 of 2

Opus max settled every doubtful size by measuring steel already in the model, such as the bent plate's legs on its twin, and checked its parts after building. Opus high's thinking caught its own errors and kept them anyway; Astra built fast but checked the brace direction in only one of its runs.

Astra, pass 2: a rear view shows the brace’s flange on the wrong edge, so it turns both braces over. In pass 1 it never looked, and the braces stayed upside down.
Astra, pass 2: a rear view shows the brace’s flange on the wrong edge, so it turns both braces over. In pass 1 it never looked, and the braces stayed upside down.
“Given time constraints, I’ll trust my earlier derivation and go with 3509.639 as the value to use.”
Opus high, pass 1 (thinking). The brace ended up 7.9 mm off.
“a 1/8″ difference is too large to ignore, so I’ll check an existing bent plate in the model”
Opus max, pass 1 (thinking). It measured the twin plate and built the right legs.

Opus max's pass 2 built all 12 parts right; its bent plate fails a very tight clash rule by 0.17 cm³.

Measured over all finished runs

Output per step
Seconds per step
Time per run
Run spent before the first part
Runs that left a part out
Opus 5.5 · max
11,060 tok
46 s
13.2 h
22%
0 of 8
Argon · high
176 tok
18 s
2.5 h
31%
1 of 15
Fable 5.1 · high
618 tok
5 s
1.4 h
43%
3 of 15
Astra · high
215 tok
10 s
1.2 h
16%
4 of 15
Opus 5.5 · high
503 tok
6 s
1.1 h
40%
4 of 15
Sol · high
319 tok
12 s
1.1 h
22%
10 of 15

Medians over finished runs; each column has its own scale. Hover a cell for its definition.

3 · Where each model fails

What the grader found wrong in each failed part. Sol and Astra most often leave a part out; Opus high most often puts parts in the wrong place or runs them into the existing steel; Opus max's only failed part in 8 runs is one clash on the stair landing.

left out
wrong place
turned or mirrored
wrong shape
holes wrong
runs into existing steel
failed parts
Opus 5.5 · max
·
·
·
·
·
1
1 of 69
Argon · high
8
10
5
3
1
3
30 of 141
Fable 5.1 · high
7
13
2
11
1
·
34 of 141
Astra · high
26
8
2
4
2
3
45 of 141
Opus 5.5 · high
9
22
·
7
2
12
52 of 141
Sol · high
25
2
3
6
3
2
41 of 141

Every failed part of every finished run, by what the grader found; the stronger the colour, the more parts. "Left out" includes parts built far from their place. On the pipe-support frame, failed posts are counted as holes wrong: the trace review found each differs from the real post only at its vent holes.

4 · Cost and time

Spend is API list price as metered by the harness. Opus max's total includes $967 of attempts lost to API usage limits and $703 on the passes we stopped; without them its finished runs cost $1,129.

Cost per finished run

$0$50$100$150$200$250$300Opus 5.5 · maxClaude Opus 5.5 (max), v5r04 pass 3: $77 · score 1.00Claude Opus 5.5 (max), v4r02 pass 1: $100 · score 1.00Claude Opus 5.5 (max), v4r07 pass 1: $104 · score 1.00Claude Opus 5.5 (max), r9 pass 3: $115 · score 1.00Claude Opus 5.5 (max), r9 pass 1: $125 · score 1.00Claude Opus 5.5 (max), r9 pass 2: $139 · score 1.00Claude Opus 5.5 (max), v5r09 pass 1: $195 · score 1.00Claude Opus 5.5 (max), v5r09 pass 2: $274 · score 0.82$120Argon · highGemini 4 Argon (high), v5r04 pass 1: $4 · score 1.00Gemini 4 Argon (high), v5r04 pass 2: $4 · score 1.00Gemini 4 Argon (high), v4r02 pass 2: $5 · score 0.60Gemini 4 Argon (high), v5r04 pass 3: $6 · score 1.00Gemini 4 Argon (high), r9 pass 1: $7 · score 1.00Gemini 4 Argon (high), v4r02 pass 3: $7 · score 1.00Gemini 4 Argon (high), v4r07 pass 3: $7 · score 1.00Gemini 4 Argon (high), v4r07 pass 1: $7 · score 0.00Gemini 4 Argon (high), v4r07 pass 2: $8 · score 0.00Gemini 4 Argon (high), r9 pass 3: $9 · score 0.54Gemini 4 Argon (high), v5r09 pass 2: $10 · score 0.83Gemini 4 Argon (high), r9 pass 2: $11 · score 1.00Gemini 4 Argon (high), v4r02 pass 1: $11 · score 0.60Gemini 4 Argon (high), v5r09 pass 1: $14 · score 0.60Gemini 4 Argon (high), v5r09 pass 3: $30 · score 1.00$7Fable 5.1 · highClaude Fable 5.1 (high), v5r04 pass 3: $17 · score 0.78Claude Fable 5.1 (high), v4r07 pass 2: $21 · score 0.32Claude Fable 5.1 (high), v4r07 pass 3: $21 · score 0.72Claude Fable 5.1 (high), v4r07 pass 1: $27 · score 1.00Claude Fable 5.1 (high), v5r04 pass 1: $28 · score 1.00Claude Fable 5.1 (high), v5r04 pass 2: $32 · score 0.06Claude Fable 5.1 (high), r9 pass 2: $35 · score 1.00Claude Fable 5.1 (high), r9 pass 3: $36 · score 1.00Claude Fable 5.1 (high), r9 pass 1: $37 · score 0.72Claude Fable 5.1 (high), v4r02 pass 2: $46 · score 1.00Claude Fable 5.1 (high), v4r02 pass 1: $48 · score 1.00Claude Fable 5.1 (high), v4r02 pass 3: $55 · score 0.00Claude Fable 5.1 (high), v5r09 pass 2: $76 · score 0.53Claude Fable 5.1 (high), v5r09 pass 3: $86 · score 0.82Claude Fable 5.1 (high), v5r09 pass 1: $92 · score 0.82$36Astra · highGPT-6 Astra (high), v5r04 pass 3: $18 · score 1.00GPT-6 Astra (high), v5r04 pass 2: $18 · score 1.00GPT-6 Astra (high), v5r04 pass 1: $22 · score 1.00GPT-6 Astra (high), v4r02 pass 1: $26 · score 0.30GPT-6 Astra (high), v4r07 pass 3: $28 · score 1.00GPT-6 Astra (high), v5r09 pass 3: $38 · score 0.45GPT-6 Astra (high), v4r02 pass 2: $42 · score 0.60GPT-6 Astra (high), v4r02 pass 3: $48 · score 1.00GPT-6 Astra (high), r9 pass 1: $55 · score 0.72GPT-6 Astra (high), v4r07 pass 1: $58 · score 0.00GPT-6 Astra (high), v4r07 pass 2: $60 · score 0.00GPT-6 Astra (high), r9 pass 2: $63 · score 0.00GPT-6 Astra (high), r9 pass 3: $63 · score 0.72GPT-6 Astra (high), v5r09 pass 1: $77 · score 0.75GPT-6 Astra (high), v5r09 pass 2: $96 · score 0.60$48Opus 5.5 · highClaude Opus 5.5 (high), v5r04 pass 2: $6 · score 1.00Claude Opus 5.5 (high), v5r04 pass 3: $8 · score 0.06Claude Opus 5.5 (high), v4r07 pass 2: $9 · score 0.36Claude Opus 5.5 (high), v4r07 pass 3: $10 · score 1.00Claude Opus 5.5 (high), v5r04 pass 1: $12 · score 0.78Claude Opus 5.5 (high), v4r07 pass 1: $15 · score 0.45Claude Opus 5.5 (high), r9 pass 1: $15 · score 0.54Claude Opus 5.5 (high), v4r02 pass 2: $18 · score 1.00Claude Opus 5.5 (high), v4r02 pass 3: $19 · score 0.60Claude Opus 5.5 (high), r9 pass 3: $21 · score 1.00Claude Opus 5.5 (high), r9 pass 2: $23 · score 0.04Claude Opus 5.5 (high), v4r02 pass 1: $27 · score 0.75Claude Opus 5.5 (high), v5r09 pass 2: $31 · score 0.56Claude Opus 5.5 (high), v5r09 pass 1: $32 · score 0.60Claude Opus 5.5 (high), v5r09 pass 3: $36 · score 0.06$18Sol · highGPT-6.1 Sol (high), v5r04 pass 1: $3 · score 0.66GPT-6.1 Sol (high), v5r04 pass 3: $3 · score 0.78GPT-6.1 Sol (high), v4r02 pass 3: $4 · score 0.30GPT-6.1 Sol (high), v5r04 pass 2: $5 · score 0.78GPT-6.1 Sol (high), v4r02 pass 2: $5 · score 0.15GPT-6.1 Sol (high), v4r07 pass 2: $5 · score 0.44GPT-6.1 Sol (high), v4r07 pass 3: $6 · score 0.44GPT-6.1 Sol (high), v4r07 pass 1: $6 · score 0.58GPT-6.1 Sol (high), v5r09 pass 2: $7 · score 0.53GPT-6.1 Sol (high), v4r02 pass 1: $7 · score 0.60GPT-6.1 Sol (high), r9 pass 2: $8 · score 0.72GPT-6.1 Sol (high), r9 pass 3: $8 · score 0.45GPT-6.1 Sol (high), v5r09 pass 3: $11 · score 0.45GPT-6.1 Sol (high), v5r09 pass 1: $11 · score 0.75GPT-6.1 Sol (high), r9 pass 1: $12 · score 0.30$6

Model spend per finished run, API list prices. Filled dot = solved run, ring = not solved, bar = median.

Cost per solved run

Claude Opus 5.5 max
$400
Gemini 4 Argon high
$17
Claude Fable 5.1 high
$109
GPT-6 Astra high
$143
Claude Opus 5.5 high
$70
GPT-6.1 Sol high
no solve

All spend of the setting (finished, stopped and set-aside runs) divided by its solved runs.

Time per finished run

04.0 h8.0 h12.0 h16.0 h20.0 h24.0 hOpus 5.5 · maxClaude Opus 5.5 (max), v5r04 pass 3: 9.1 h · score 1.00Claude Opus 5.5 (max), v4r02 pass 1: 10.4 h · score 1.00Claude Opus 5.5 (max), v4r07 pass 1: 10.5 h · score 1.00Claude Opus 5.5 (max), r9 pass 3: 12.8 h · score 1.00Claude Opus 5.5 (max), r9 pass 1: 13.5 h · score 1.00Claude Opus 5.5 (max), r9 pass 2: 14.0 h · score 1.00Claude Opus 5.5 (max), v5r09 pass 1: 17.5 h · score 1.00Claude Opus 5.5 (max), v5r09 pass 2: 25.9 h · score 0.8213.2 hArgon · highGemini 4 Argon (high), v5r04 pass 2: 1.7 h · score 1.00Gemini 4 Argon (high), v5r04 pass 1: 2.2 h · score 1.00Gemini 4 Argon (high), v4r07 pass 3: 2.3 h · score 1.00Gemini 4 Argon (high), r9 pass 2: 2.4 h · score 1.00Gemini 4 Argon (high), r9 pass 3: 2.4 h · score 0.54Gemini 4 Argon (high), v4r07 pass 1: 2.5 h · score 0.00Gemini 4 Argon (high), v4r02 pass 2: 2.5 h · score 0.60Gemini 4 Argon (high), v4r07 pass 2: 2.5 h · score 0.00Gemini 4 Argon (high), r9 pass 1: 2.5 h · score 1.00Gemini 4 Argon (high), v5r04 pass 3: 2.7 h · score 1.00Gemini 4 Argon (high), v4r02 pass 3: 3.6 h · score 1.00Gemini 4 Argon (high), v5r09 pass 2: 3.7 h · score 0.83Gemini 4 Argon (high), v4r02 pass 1: 3.8 h · score 0.60Gemini 4 Argon (high), v5r09 pass 1: 4.1 h · score 0.60Gemini 4 Argon (high), v5r09 pass 3: 5.1 h · score 1.002.5 hFable 5.1 · highClaude Fable 5.1 (high), v5r04 pass 3: 42 min · score 0.78Claude Fable 5.1 (high), v4r07 pass 2: 47 min · score 0.32Claude Fable 5.1 (high), v4r07 pass 1: 53 min · score 1.00Claude Fable 5.1 (high), v4r07 pass 3: 53 min · score 0.72Claude Fable 5.1 (high), v5r04 pass 1: 1.1 h · score 1.00Claude Fable 5.1 (high), r9 pass 2: 1.2 h · score 1.00Claude Fable 5.1 (high), v5r04 pass 2: 1.3 h · score 0.06Claude Fable 5.1 (high), r9 pass 1: 1.4 h · score 0.72Claude Fable 5.1 (high), r9 pass 3: 1.5 h · score 1.00Claude Fable 5.1 (high), v4r02 pass 2: 1.7 h · score 1.00Claude Fable 5.1 (high), v4r02 pass 1: 1.9 h · score 1.00Claude Fable 5.1 (high), v4r02 pass 3: 2.3 h · score 0.00Claude Fable 5.1 (high), v5r09 pass 2: 3.6 h · score 0.53Claude Fable 5.1 (high), v5r09 pass 1: 3.6 h · score 0.82Claude Fable 5.1 (high), v5r09 pass 3: 4.3 h · score 0.821.4 hAstra · highGPT-6 Astra (high), v5r04 pass 2: 19 min · score 1.00GPT-6 Astra (high), v5r04 pass 3: 19 min · score 1.00GPT-6 Astra (high), v5r04 pass 1: 20 min · score 1.00GPT-6 Astra (high), v4r07 pass 3: 38 min · score 1.00GPT-6 Astra (high), v4r02 pass 1: 46 min · score 0.30GPT-6 Astra (high), v5r09 pass 3: 54 min · score 0.45GPT-6 Astra (high), v4r02 pass 2: 1.1 h · score 0.60GPT-6 Astra (high), r9 pass 1: 1.2 h · score 0.72GPT-6 Astra (high), v4r02 pass 3: 1.2 h · score 1.00GPT-6 Astra (high), v4r07 pass 2: 1.3 h · score 0.00GPT-6 Astra (high), r9 pass 2: 1.4 h · score 0.00GPT-6 Astra (high), r9 pass 3: 1.4 h · score 0.72GPT-6 Astra (high), v4r07 pass 1: 1.4 h · score 0.00GPT-6 Astra (high), v5r09 pass 1: 1.7 h · score 0.75GPT-6 Astra (high), v5r09 pass 2: 2.2 h · score 0.601.2 hOpus 5.5 · highClaude Opus 5.5 (high), v5r04 pass 2: 23 min · score 1.00Claude Opus 5.5 (high), v5r04 pass 3: 29 min · score 0.06Claude Opus 5.5 (high), v4r07 pass 2: 38 min · score 0.36Claude Opus 5.5 (high), v5r04 pass 1: 47 min · score 0.78Claude Opus 5.5 (high), v4r07 pass 3: 48 min · score 1.00Claude Opus 5.5 (high), v4r07 pass 1: 1.1 h · score 0.45Claude Opus 5.5 (high), v4r02 pass 2: 1.1 h · score 1.00Claude Opus 5.5 (high), r9 pass 1: 1.1 h · score 0.54Claude Opus 5.5 (high), v4r02 pass 3: 1.3 h · score 0.60Claude Opus 5.5 (high), v5r09 pass 3: 1.6 h · score 0.06Claude Opus 5.5 (high), v4r02 pass 1: 1.7 h · score 0.75Claude Opus 5.5 (high), v5r09 pass 1: 1.7 h · score 0.60Claude Opus 5.5 (high), v5r09 pass 2: 1.8 h · score 0.56Claude Opus 5.5 (high), r9 pass 2: 2.0 h · score 0.04Claude Opus 5.5 (high), r9 pass 3: 2.2 h · score 1.001.1 hSol · highGPT-6.1 Sol (high), v5r04 pass 1: 24 min · score 0.66GPT-6.1 Sol (high), v5r04 pass 3: 29 min · score 0.78GPT-6.1 Sol (high), v5r04 pass 2: 46 min · score 0.78GPT-6.1 Sol (high), v4r02 pass 3: 53 min · score 0.30GPT-6.1 Sol (high), v4r07 pass 2: 58 min · score 0.44GPT-6.1 Sol (high), v4r07 pass 3: 1.1 h · score 0.44GPT-6.1 Sol (high), v4r02 pass 2: 1.1 h · score 0.15GPT-6.1 Sol (high), v4r07 pass 1: 1.1 h · score 0.58GPT-6.1 Sol (high), v5r09 pass 2: 1.4 h · score 0.53GPT-6.1 Sol (high), r9 pass 2: 1.5 h · score 0.72GPT-6.1 Sol (high), v4r02 pass 1: 1.5 h · score 0.60GPT-6.1 Sol (high), r9 pass 3: 1.6 h · score 0.45GPT-6.1 Sol (high), v5r09 pass 3: 2.3 h · score 0.45GPT-6.1 Sol (high), v5r09 pass 1: 2.3 h · score 0.75GPT-6.1 Sol (high), r9 pass 1: 2.3 h · score 0.301.1 h

Time from the first to the last step of the graded try. Filled dot = solved run, ring = not solved, bar = median.

5 · The five tasks

All five are rebuilds: a real steel model with pieces removed, the shop and erection drawings of those pieces, and the FreeCAD desktop driven by screenshots, mouse and keyboard only. Red marks what was removed.

The whole model of task v4r07, removed steel in redClose-up of the removed steel in red

1 · Skewed rooftop pipe-support frame

v4r07 · rebuild · 10 parts · drawings: 1 shop, 2 erection
Removed
2 square-tube girts (HSS 4x3) and the 8 bent "knife" plates that weld them to the posts, in a bay set at an angle to the grid
The prompt
Rebuild the 10 parts in place. They were welded on site, so no open holes show where they go: the position comes from an enlarged plan of the skewed bay and the sections.
The trap
The skew. Every dimension is along the skewed bay, not the building grid, and there are no holes to snap to.
Opus max
✓
Argon
.00.00✓
Fable
✓.32.72
Astra
.00.00✓
Opus high
.45.36✓
Sol
.58.44.44
The whole model of task v4r02, removed steel in redClose-up of the removed steel in red

2 · Double-angle X-bracing in a factory wall

v4r02 · rebuild · 6 parts · drawings: 2 shop, 1 erection
Removed
the 2 lower diagonal braces of an X-braced bay, each made of 2 angles (L5x3x1/4) back to back, and their 2 filler plates
The prompt
Rebuild the 6 parts in place with every bolt hole. The open holes in the existing gusset plates show where the braces connect.
The trap
Brace length. It must come from the bill of material and the end holes; the rounded work-point length pushes the angles 1/16 in into the gusset (a 1.9 cm³ clash).
Opus max
✓
Argon
.60.60✓
Fable
✓✓.00
Astra
.30.60✓
Opus high
.75✓.60
Sol
.60.15.30
The whole model of task v5r04, removed steel in redClose-up of the removed steel in red

3 · Stair stringer with its tread angles

v5r04 · rebuild · 14 parts · drawings: 1 shop, 4 erection
Removed
one channel stringer (C12x20.7) of an exit stair, with its 7 small angles, 4 flat bars and plates, and a round bar
The prompt
Rebuild the 14 parts in place, on the slope, from one shop sheet and four erection sheets.
The trap
Many small parts on a slope. It is easy to stop with one small plate missing.
Opus max
✓
Argon
✓✓✓
Fable
✓.06.78
Astra
✓✓✓
Opus high
.78✓.06
Sol
.66.78.78
The whole model of task r9, removed steel in redClose-up of the removed steel in red

4 · Pipe-support frame on a pipe rack

r9 · rebuild · 5 parts · drawings: 1 shop, 2 erection
Removed
one shop-welded frame: a W10x22 beam with 16 bolt holes on two short W6x15 legs, with 2 heavy plates
The prompt
Rebuild the 5 parts in place. The open slots in the existing angles show where the beam bolts on. The beam's section has no profile file; the identical beams in the model show it.
The trap
Small details: the beam must sit exactly against the existing angles, and each short post has small vent holes in a detail that is easy to miss.
Opus max
✓✓✓
Argon
✓✓.54
Fable
.72✓✓
Astra
.72.00.72
Opus high
.54.04✓
Sol
.30.72.45
The whole model of task v5r09, removed steel in redClose-up of the removed steel in red

5 · Stair landing post frames

v5r09 · rebuild · 12 parts · drawings: 3 shop, 5 erection
Removed
3 square-tube posts (HSS 4x4), 2 sloped angle braces, 3 clip angles, 3 small plates and 1 bent plate under a stair landing
The prompt
Rebuild the 12 parts in place from three shop sheets and five erection sheets.
The trap
The widest spread of mistakes: clip angles turned 180°, a bent plate drawn as its mirror image, and parts clashing with the landing steel.
Opus max
✓.82
Argon
.60.83✓
Fable
.82.53.82
Astra
.75.60.45
Opus high
.60.56.06
Sol
.75.53.45

6 · How we ran it

SettingRunsWhenRoute and limits
Claude Opus 5.5 · maxstudy, 3 per task29 Oct - 2 Nov 2026Anthropic API, effort max, summarized thinking returned, 128k reply cap
GPT-6 Astra · highstudy, 3 per task29 - 30 Oct 2026OpenAI API, reasoning effort high (no reasoning text returned)
Claude Fable 5.1 · highstudy, 3 per task29 - 30 Oct 2026Anthropic API, effort high, summarized thinking returned
Claude Opus 5.5 · highearlier, 3 per taskOct 2026Anthropic API, effort high
GPT-6.1 Sol · highearlier, 3 per taskOct 2026OpenAI API, reasoning effort high
Gemini 4 Argon · highearlier, 3 per taskOct 2026Google model by API, reasoning effort high

Appendix · every run

TaskModelPassScorePartsStepsTimeCostWhat the grader found
v4r07Opus max max11.0 ✓10/1032410.5 h$104.15—
v4r07Opus max max2——285—$123.45stopped before it finished
v4r07Opus max max3——323—$119.50stopped before it finished
v4r07Argon high10.000/104422.5 h$7.47left out, wrong place
v4r07Argon high20.000/104102.5 h$8.16turned or mirrored, wrong place
v4r07Argon high31.0 ✓10/104082.3 h$7.23—
v4r07Fable high11.0 ✓10/1037953 min$26.75—
v4r07Fable high20.326/1027847 min$20.72left out
v4r07Fable high30.728/1025153 min$20.74wrong shape
v4r07Astra high10.000/104431.4 h$58.21left out, wrong place
v4r07Astra high20.000/104591.3 h$59.54left out, wrong place
v4r07Astra high31.0 ✓10/1023638 min$27.95—
v4r07Opus high high10.455/103551.1 h$14.52wrong place
v4r07Opus high high20.364/1023638 min$9.12wrong place, wrong shape
v4r07Opus high high31.0 ✓10/1023648 min$10.34—
v4r07Sol high10.588/103471.1 h$5.95left out
v4r07Sol high20.447/1030358 min$5.21left out
v4r07Sol high30.447/103301.1 h$5.86left out
v4r02Opus max max11.0 ✓6/633710.4 h$99.65—
v4r02Opus max max2——261—$138.39stopped before it finished
v4r02Opus max max3——314—$115.97stopped before it finished
v4r02Argon high10.604/66943.8 h$10.99wrong shape
v4r02Argon high20.604/64492.5 h$5.23runs into existing steel
v4r02Argon high31.0 ✓6/66383.6 h$7.02—
v4r02Fable high11.0 ✓6/64991.9 h$48.00—
v4r02Fable high21.0 ✓6/65351.7 h$46.39—
v4r02Fable high30.000/65682.3 h$54.75wrong shape
v4r02Astra high10.302/621146 min$26.41left out
v4r02Astra high20.604/63201.1 h$42.50runs into existing steel
v4r02Astra high31.0 ✓6/63611.2 h$48.33—
v4r02Opus high high10.755/65671.7 h$27.03runs into existing steel
v4r02Opus high high21.0 ✓6/63751.1 h$17.63—
v4r02Opus high high30.604/63971.3 h$18.98runs into existing steel
v4r02Sol high10.604/63851.5 h$7.40runs into existing steel
v4r02Sol high20.151/62481.1 h$4.73left out, wrong place
v4r02Sol high30.302/623953 min$4.45left out
v5r04Opus max max1——308—$112.55stopped before it finished
v5r04Opus max max2——252—$93.56stopped before it finished
v5r04Opus max max31.0 ✓14/144919.1 h$77.02—
v5r04Argon high11.0 ✓14/142672.2 h$3.76—
v5r04Argon high21.0 ✓14/142361.7 h$4.22—
v5r04Argon high31.0 ✓14/143532.7 h$5.56—
v5r04Fable high11.0 ✓14/143741.1 h$27.96—
v5r04Fable high20.061/144161.3 h$32.43wrong place
v5r04Fable high30.7813/1423242 min$16.66left out
v5r04Astra high11.0 ✓14/1416220 min$21.94—
v5r04Astra high21.0 ✓14/1415619 min$18.23—
v5r04Astra high31.0 ✓14/1414519 min$17.65—
v5r04Opus high high10.7813/1431747 min$12.28left out
v5r04Opus high high21.0 ✓14/1416223 min$6.13—
v5r04Opus high high30.061/1420729 min$7.51wrong place
v5r04Sol high10.6612/1418024 min$2.58left out
v5r04Sol high20.7813/1426046 min$4.56left out
v5r04Sol high30.7813/1420329 min$3.06left out
r9Opus max max11.0 ✓5/551513.5 h$125.41—
r9Opus max max21.0 ✓5/544814.0 h$138.65—
r9Opus max max31.0 ✓5/541612.8 h$114.56—
r9Argon high11.0 ✓ (was 0.72)5/55392.5 h$6.99—
r9Argon high21.0 ✓5/55852.4 h$10.90—
r9Argon high30.543/54772.4 h$8.90wrong shape
r9Fable high10.72 (was 0.54)4/54751.4 h$36.56wrong shape
r9Fable high21.0 ✓ (was 0.72)5/54771.2 h$35.35—
r9Fable high31.0 ✓ (was 0.72)5/54321.5 h$35.97—
r9Astra high10.724/54321.2 h$55.39wrong shape
r9Astra high20.000/54621.4 h$62.94wrong place, wrong shape
r9Astra high30.72 (was 0.54)4/54761.4 h$63.45wrong shape
r9Opus high high10.54 (was 0.36)3/53591.1 h$15.35wrong shape
r9Opus high high20.04 (was 0.00)1/55542.0 h$22.79left out
r9Opus high high31.0 ✓ (was 0.72)5/54652.2 h$21.17—
r9Sol high10.302/56012.3 h$12.40turned or mirrored, wrong shape
r9Sol high20.724/54431.5 h$8.00left out
r9Sol high30.453/54681.6 h$8.17wrong shape
v5r09Opus max max11.0 ✓12/1243017.5 h$194.93—
v5r09Opus max max20.8211/1257325.9 h$274.33runs into existing steel
v5r09Opus max max3——0—$0.00stopped before it finished
v5r09Argon high10.608/126704.1 h$14.05runs into existing steel, turned or mirrored
v5r09Argon high20.8312/126113.7 h$9.66—
v5r09Argon high31.0 ✓12/128875.1 h$29.67—
v5r09Fable high10.8211/129673.6 h$91.56wrong shape
v5r09Fable high20.537/126733.6 h$76.26left out, turned or mirrored, wrong shape
v5r09Fable high30.8211/127214.3 h$85.71turned or mirrored
v5r09Astra high10.7510/125801.7 h$76.70turned or mirrored
v5r09Astra high20.608/126482.2 h$95.75runs into existing steel, wrong shape
v5r09Astra high30.456/1229654 min$37.66left out
v5r09Opus high high10.608/127081.7 h$32.29runs into existing steel, wrong shape
v5r09Opus high high20.569/126481.8 h$30.84left out, runs into existing steel
v5r09Opus high high30.061/127271.6 h$35.65left out, runs into existing steel, wrong shape
v5r09Sol high10.7510/126152.3 h$11.41turned or mirrored, wrong place
v5r09Sol high20.538/124111.4 h$7.23left out
v5r09Sol high30.456/125782.3 h$10.65turned or mirrored, wrong shape