More reasoning did not mean better work.
Across the valid common tasks, medium matched every higher reasoning setting while taking less time and fewer input tokens.
294 measured runs across GPT-5.6 Luna, Terra, and Sol through ChatGPT subscription access in Codex.
Published 01 Aug 2026
How the evidence changed
- 294measured rows retained in the historical ledger
- 50rows excluded after two defective checks were found
- 210rows eligible for the corrected common policy view
The decision
Start ordinary bounded work at medium. Escalate only when the task names a hard problem, costly failure, or unresolved effort decision.
- 42/42
- accepted at medium, high, xhigh, and max
- No winner
- the valid tasks did not separate Luna, Terra, and Sol
- Medium first
- verify the work before paying for more reasoning
The result stayed flat. The cost did not.
Follow the same accepted result through each reasoning level. The bars show mean time, while the figures preserve the measured token cost.
Accepted work stays level while time and input tokens rise.
Medium reached the measured ceiling
Medium accepted every valid common task. This is the first complete result in the comparison.
High added cost, not accepted work
High also accepted 42 of 42, but required 37.54 seconds and 58,591 input tokens per accepted result.
Extra High added cost, not accepted work
Extra High also accepted 42 of 42, but required 47.17 seconds and 66,298 input tokens per accepted result.
Max added cost, not accepted work
Max also accepted 42 of 42, but required 57.11 seconds and 72,143 input tokens per accepted result.
The operating policy
Codex presents reasoning as a dial. Turning that dial upward sounds safer: give the model more time, get better work. It also means longer waits and greater usage. Without evidence, a team can spend more on every task without knowing whether the result changed.
The benchmark was built to answer an everyday business question: which Codex setting should a team use before it knows whether a task is hard?
Our answer is a starting policy, not a promise:
| Situation | Start here | Why |
| Ordinary bounded work with clear checks | Any available GPT-5.6 model at medium | Medium matched the higher settings on every valid common task while using less time and fewer tokens. |
| Difficult evaluation, code review, or adversarial checking | Sol high as a scout | Sol high is a reasonable hypothesis for careful evaluation, but it did not beat medium in this benchmark. |
| A task where deeper reasoning itself is the unresolved decision | Compare the smallest relevant pair | Use xhigh or max only when the outcome could change a real routing choice. |
| A task that divides into independent workstreams |
What we tested
We compared three GPT-5.6 models available through Codex with ChatGPT subscription access:
- Luna
- Terra
- Sol
For a fair common comparison, every model received the same frozen task, prompt, tools, fixture, and acceptance checks at five reasoning levels:
| Desktop wording | Runtime value | Plain-language meaning in this study |
| Light | low | Less reasoning, usually faster |
| Medium | medium | Balanced starting level |
| High | high | More reasoning for harder work |
| Extra High | xhigh | A larger reasoning allowance |
| Not shown in the Sol desktop picker |
What the results actually say
The first result looked simple. Sol led the original common pass. The careful result was different.
Two task families had invalid evaluation rules and had to be removed from model routing conclusions. After excluding those rows, the valid common view was:
| Model | Accepted valid rows | What this supports |
| Luna | 69/70 | Strong performance on these tasks, with one repeated low-effort failure |
| Terra | 70/70 | Complete acceptance on these tasks |
| Sol | 70/70 | Complete acceptance on these tasks |
Terra and Sol tied. Luna differed only on one exact low-effort implementation task. That is not enough to name a general winner.
The more important finding is that the suite was mostly too easy for these models. When several options pass almost everything, the benchmark confirms a competence floor but stops helping with selection. A perfect score can mean the model is excellent. It can also mean the test reached its ceiling.
Medium matched the higher settings
Across the valid common rows, reasoning effort produced this result:
A confident score measured the wrong thing.
The page could remain correct while the old checker changed its verdict. This is why the historical frontend rows cannot select a model.
The page rendered as intended
Desktop showed two readable regions and mobile stacked them. The user-facing outcome was correct.
A harmless wrapper changed the score
Adding a semantic container preserved the same layout, but the old checker inspected only direct children and reported failure.
A repaired check reversed most failures
The apparent aggregate moved from 7 of 33 under the old rule to 28 of 33 under a prototype repair.
The historical rows were quarantined
All 35 PUB8 rows remain in the ledger but no longer support routing. PUB8V2 is separately versioned and not yet measured on a model.
Two benchmark failures changed the story
The benchmark itself failed in two useful ways. Correcting those failures changed the apparent result, which is why they belong in the main story.
One task asked for five stages but graded four
The public workflow task called PUB4 required five stages in its written task
card. Its automated check expected four semantic stages. A model could follow
the instructions and still fail the grader.
All 15 PUB4 quality rows were excluded. We did not quietly rewrite the scores.
The historical outputs remain preserved, but they cannot support a model or
reasoning-level claim.
The responsive-page check confused valid layouts with failures
The original responsive frontend task, PUB8, tried to infer a two-column
desktop layout by inspecting direct children in the page structure. A valid
implementation with an extra semantic wrapper could render correctly and still
fail. Padding could also be mistaken for a column.
A later sensitivity check changed the apparent aggregate result from 7/33
under the old rule to 28/33 under a prototype repair. That large change shows
that the checker was driving much of the result.
All 35 historical PUB8 rows were quarantined from routing evidence. We built a
separately versioned replacement,
What each audience should take away
Engineering leaders
Start bounded work at medium and require tests, diffs, screenshots, or other external proof. Reserve higher reasoning for tasks where failure is costly or where medium has already shown a concrete limitation.
This recommendation applies to work resembling the tested tasks. It does not establish a default for security-critical systems, production migrations, large-scale architecture, or unfamiliar repositories with weak tests.
Product and operations teams
Do not equate a higher reasoning label with a better business outcome. Measure the time and human effort required to reach an accepted result. A cheaper first attempt that creates review work may cost more overall. A slower setting that changes nothing may also be waste.
The benchmark measured task acceptance, latency, and exposed token use. It did not measure long-term maintenance cost, customer impact, or the value of a production deployment.
Developers using Codex every day
Use medium as the starting point, then escalate deliberately:
- Write down what must be true when the task is finished.
- Run the task at medium.
- Check the result with evidence outside the model's own explanation.
- Move to high or above only when the first attempt exposes a real reasoning gap or the cost of failure justifies more scrutiny.
For frontend work, inspect deterministic desktop and mobile screenshots. Do not
use the historical PUB8 results to choose a model.
Researchers and benchmark builders
Separate model failure from test failure. Preserve raw results, but version a repaired task instead of rewriting history. Repeat only cells that could change a decision. When every lane passes, build a more discriminating task only if a real routing question needs it.
These practices should generalize to many model evaluations. The exact scores and routing advice do not.
What we still do not know
This study leaves several questions open:
- whether Sol high outperforms medium on naturally difficult repository work
- whether full desktop Ultra delegation helps tasks that split cleanly into independent streams
- which model is best for responsive frontend implementation under the new
PUB8V2check - whether the same behavior appears through paid API access
- whether future Codex versions, tools, or model updates change the result
- how the models compare on long-lived production work where maintainability appears weeks later
Those unknowns are not defects to hide. They define where the evidence stops.
The decision in one minute
- Start at medium. It matched every higher common reasoning level on the valid tasks while using less time and fewer tokens.
- Do not name a universal model winner. Luna, Terra, and Sol were nearly indistinguishable after invalid tasks were removed.
- Use Sol high as a scout, not a default winner. It is a reasonable lane for difficult evaluation and review, but it did not beat medium here.
- Gate xhigh, max, and ultra. Use them when a named task risk or blocked decision can justify the additional cost.
- Verify the work externally. Tests, browser checks, and evidence matter more than the label in the model picker.
- Audit the benchmark when a result looks surprising. Two defective checks materially changed this study's first interpretation.
The next useful experiment is not another broad sweep. It is one naturally occurring hard task with a clear contract, a preselected model and effort, and an outcome that can be checked independently. Only then can a new measured row resolve a real decision instead of adding volume.
Method and scope
The benchmark was completed in July 2026 through Codex ChatGPT-subscription
OAuth using Codex app CLI 0.144.2. It used frozen private and public task
fixtures, a common core-tool profile, isolated execution state, and requested
versus observed identity checks. The retained evidence contains 294 measured
runs. Fifty historical rows are not current routing evidence: 15 from the
contradictory PUB4 task and 35 from the quarantined PUB8 visual check.
The corrected common policy view contains 210 eligible rows. It applies only to the tested model versions, reasoning levels, tools, fixtures, and July 2026 runtime. It does not establish paid API behavior, universal coding ability, production safety, future model behavior, or a global model ranking.
The public benchmark package has been prepared for reproducibility but has not been published. Private fixtures and raw traces remain local because they can contain environment-specific paths and context.
Product semantics were checked against OpenAI's
GPT-5.6 model guidance
and GPT-5.6 announcement, accessed 1 August 2026. OpenAI describes max as additional time for exploration, checking, and
revision. It describes ultra as a multi-agent setting that coordinates four
agents in parallel by default. Product availability and behavior can change,
so current users should check the latest documentation and their own runtime.