1) Use purpose-built test harnesses We use Claude Code or Codex
2) Test one-shot tasks We work in sessions
Sessions are messy, we start with a large primary task, then some cleanup, an adjacent fix here another over there. We start, stop, and change our minds.
That changes both cost and quality. Cache TTLs expire. Context grows.
I am working on creating one. Here's my rough plan: 1) Use Claude Code and Codex 2) Use session shaped workloads. Stitch multiple SWE bench verified tasks into one big session. 2.a) Use tasks from the same repo to ensure topical continuity. 3) Headline metrics: dollar cost vs quality. 3.a) Secondary metrics: turn count, time to completion
Open Questions a) Does this problem make sense? b) Does my benchmark spec make sense? c) Are 10 SWE bench verified task starting with a hard one the right workload shape?
At a high level, it
- Mines tasks from your merged PRs/commits - Replays them in Docker containers with different harness settings (change model / reasoning effort / AGENTS.md / etc) - Grades the patches on various attributes (tests, equivalence with human patch, code quality)
The goal is to get a sense of how agents perform on your tasks, with your context, using the tools you do.
This is currently one-shot but I'd definitely like to explore session-based benchmarks as well. There are some interesting papers that just came out on this https://arxiv.org/abs/2606.29957 https://arxiv.org/abs/2606.30573
The only big gap left is that they aren't using claude code/codex as harnesses. I'll try to reuse their constructed user sessions.
PS: Your work at stet is also interesting. That's definitely a problem right now that's hard to track. The only real solution is more robust CI/CD. I have since added harder validation like essentially running a full benchmark run on every prod push.
Each "task" is the equivalent of a new user query and we also pre-program "cache expiration" (sleep for 5 mins) into the session. This ensures parity across providers (both default to 5 min TTLs).
The goal of this exercise is to tease out how Claude Code and Codex differ in managing their context and how that impacts cost and quality for the same simulated session.