Latest research · Jackbench
Harness choice moved a local model 19.4 points
Qwen 3.6 27B solved 103 of 144 tasks in Pi and 75 of 144 in OpenCode. Nothing changed between the runs except the harness, and the grading is deterministic. It runs on the studio's own hardware, so the provider cost was nothing. The write-up covers the hosted baseline too, including the effects that didn't survive correction.
+19.4
points, Pi over OpenCode
Pi103/144 · 71.5%
OpenCode75/144 · 52.1%
95% CI10.4–28.5
exact McNemarp = 6.2e-05
provider cost0 USD
The studio
Cold Anvil Studios is a one-person AI research and development studio. I'm Jack. I was a product manager for fifteen years; these days I build AI systems and run benchmark studies, with a fleet of agents doing most of the engineering. The research gets published here.
About the studioCurrent projects
All projects
Active
Jackbench
A preregistered study of how the choice of coding harness changes model results, with deterministic grading and sealed per-attempt receipts.
In Development
The studio harness
An agentic engineering harness with a governed run lifecycle, independent plan checking and two-bar verification; the intent is to open-source it once it has been evaluated properly.
Concept
Beararms.ai
A reserved name and some notes. Nothing public yet.