KeelenBenchmark: engines on an identical spec
Same spec, three engines. The output matched. The cost did not.
12 runs, 2026-07-20 to 2026-08-10 · Updated 20 August 2026
Across 12 controlled runs we gave Claude Code, OpenAI Codex, and Moonshot Kimi the same freshly written specification, the same starter repository, and the same merge gates, then measured what each one shipped. All three delivered the same median number of merged pull requests: three. What separated them was consumption. Claude Code used about 4.7 times the input tokens of Codex for that same result, and Kimi about 2.4 times. Differences in speed and iteration count were smaller than the variation between individual runs, so we do not read them as a ranking.
The practical consequence for anyone choosing an engine: pick on cost and on the plan you already own, not on the assumption that one of them cannot do the work. In this test they all did the work.
What we measured
Medians across the runs that recorded each metric. The sample size differs per metric because older runs did not record every field, so each cell carries its own n. Token columns are relative to Codex, the lowest consumer.
| Engine | Merged PRs | Iterations | Min to 1st merge | Input tokens | Output tokens |
|---|---|---|---|---|---|
| Claude Code12 runs | 3n=7 | 8n=7 | 33n=8 | 12.1M · 4.7xn=9 | 166k · 5.7xn=9 |
| OpenAI Codex12 runs | 3n=7 | 10n=7 | 38n=8 | 2.6M · 1xn=9 | 29k · 1xn=9 |
| Moonshot Kimi11 runs | 3n=6 | 12.5n=6 | 43n=7 | 6.3M · 2.4xn=8 | 90k · 3.1xn=8 |
Read the merged-PR column first: it is the same for all three. The iteration and timing columns differ by less than the spread between individual runs of the same engine, so we treat them as noise rather than as a ranking. The token columns are the one difference large and consistent enough to act on.
How the test works
One spec, invented fresh each run
Every run starts from a specification written for that run, so no engine can have seen it before. All three engines receive the same text.
Three brand-new repositories
Each engine gets its own newly created project of the same kind, starting from the same scaffold. Nothing is shared between them and nothing carries over from a previous run.
The real product loop, not a harness
Each project runs the same autonomous loop customers get: planning, dev iterations, the five merge gates, real CI, and real pull requests. There is no benchmark-specific shortcut.
Measured from the database, not from impressions
Merged pull requests, iteration counts, wall-clock timings, and token usage are read from the run records rather than judged by a model or by us.
Curated by hand before publication
The figures on this page are a fixed snapshot taken on 2026-08-20. The page is not wired live to the database, so it cannot publish anything a later run happens to record.
What this does not show
- Twelve runs is a small sample, and the `n` beside each number is smaller still for metrics that older runs did not record. Treat every figure as directional.
- The projects are small and newly created. These numbers say nothing about behaviour on a large existing codebase.
- We ran Python libraries and web apps. Other stacks may differ.
- The engines were tested as we drive them, inside our loop and our prompts. A different harness would produce different numbers, so this is not a general claim about any vendor's model.
- Engines improve on their own schedule. A figure from this window will not hold forever, and the date above is the honest scope of it.
- We deliberately do not rank the engines on quality. Twelve runs cannot support that claim about anybody's product.
Use the engine you already pay for
Keelen runs on credentials you connect, and the engine is a per-project setting you can change whenever you like.
FAQ
Which AI engine is best for coding?
In this test, none of them was distinguishable on output: Claude Code, Codex, and Kimi each shipped a median of three merged pull requests from the same specification. They differed on consumption, where Codex was the cheapest by a wide margin. Choose on the plan you already pay for and on token cost, not on an assumption that one of them cannot do the job.
Why does Claude Code use so many more tokens?
Most of the gap is input tokens, which is context being re-sent, rather than more code being written. That is a property of how the work is packaged rather than of the code produced, which is why the output was equivalent. It also means the gap can move as tooling changes.
Does the token difference cost me more money?
It depends on what you connected. Keelen runs on your own credentials, so usage lands on your provider account at cost. On a flat-rate subscription a heavier engine mostly consumes your allowance faster; on a metered API key it shows up on the bill.
Can I change engine per project?
Yes. The engine is a per-project setting, so you can point one project at the subscription you already own and another at a cheaper key, and change your mind later.
Why is GLM not in this comparison?
Keelen supports Zhipu GLM, but these particular runs covered Claude Code, Codex, and Kimi only. We would rather leave a row out than publish a number we did not measure.