Same spec, three engines. Consumption differed.

What we measured

Medians across the runs that recorded each metric. The sample size differs per metric, and some older runs did not record every field. Each cell shows its own n. Token columns compare with Codex, which used the least.

Start with the merged-PR column. It is the same for all three engines. The iteration and timing columns differ less than the spread between runs of the same engine. So we treat them as noise. We do not read them as a ranking. The token columns hold the one difference large enough to act on.

How the test works

1. 01

   One spec, invented fresh each run

   We write a new spec for every run. No engine has seen it before. All three engines get the same text.

2. 02

   Three brand-new repositories

   Each engine gets its own new project of the same kind. Each one starts from the same scaffold. Nothing is shared. Nothing carries over from an earlier run.

3. 03

   The real product loop

   Each project runs the same loop our customers get. That means planning, dev iterations, the five merge gates, real CI, and real pull requests. There is no shortcut for this test.

4. 04

   Measured from the run records

   We read merged pull requests, iteration counts, wall clock times, and token use from the run records. No model and no person scores them.

5. 05

   Curated by hand before publication

   This page is a fixed snapshot taken on 2026-08-20. It is not wired to the database. It cannot show what a later run records.

6. 06

   Capacity limits stay on the record

   An engine can stop early if it runs out of its provider allowance. We record those runs as their own outcome. They do not count as a failure.

7. 07

   An uneven roster holds the page

   We publish an engine only when it ran the same specs as the others. We could refresh some rows. That would raise the sample for some engines and leave others behind. Each number sits beside its own sample size. So a gap would read as a ranking. An uneven set of rows is a reason to hold this snapshot.

What this does not show

• ×Twelve runs is a small sample. The `n` beside a number is often smaller. Some older runs did not record every field. Treat each figure as a rough guide.
• ×The projects are small and new. These numbers say nothing about a large or old repo.
• ×We ran Python libraries and web apps. Other stacks may differ.
• ×We ran the engines the way we drive them, inside our loop and our prompts. A different harness would give other numbers. So this is not a general claim about any vendor's model.
• ×Engines improve on their own schedule. A figure from this window will not hold forever, and the date above is the honest scope of it.
• ×Token counts and merged PR counts are not billed cost. They are not accepted work or a quality score. Twelve runs cannot support those claims about anyone's product.