MaxxBench: Which Coding Models Follow Changed Requirements?

MaxxBench changes 89 DeepSWE tasks to test adaptation. Compare the original 12 model configurations and reasoning settings, understand failures caused by guessed hidden tests, choose an access route, and run a focused requirement-change trial in your repository.

Contents
MaxxBench: Which Coding Models Follow Changed Requirements?

MaxxBench matters when similar DeepSWE scores make coding agents look interchangeable, but your agent ignores a revised requirement. For that workload, shortlist Claude Opus 5.5 and GPT-6 Astra first, based on unLAP’s reported results. Test both against one changed requirement in your repository before changing your default model.

Which models should you test first?

Claude Opus 5.5 leads the reported comparison, with GPT-6 Astra and GPT-5.6 Sol next. Treat that order as a shortlist for specification-sensitive work, not a guarantee about your coding tool.

The table preserves the original 12-configuration comparison. As of 2026-11-16, the live research comparison includes 17 configurations. Scores come from the research table; reasoning settings come from the MaxxBench leaderboard.

ModelReasoning effortPublished DeepSWEMaxxBench
Claude Opus 5.5max74.2%76.4%
GPT-6 Astraxhigh74.1%71.9%
GPT-5.6 Solmax72.7%69.1%
Claude Fable 5xhigh69.9%63.5%
Claude Opus 5max73.6%63.5%
GLM-5.3max69.0%56.7%
DeepSeek V4.1 Flashmax74.2%55.6%
GLM-5.3 Flashmax63.4%55.1%
Gemini 3.8 Flashhigh73.8%53.4%
GPT-5.6 Lunamax67.2%46.1%
Muse Spark 1.2xhigh54.9%38.8%
DeepSeek V4 Flashmax53.3%28.1%

Within these 12 configurations, the score range expands from 20.9 percentage points on DeepSWE to 48.3 on MaxxBench. Six configurations originally within 1.5 points spread across 23.0 points. Opus 5.5 is the only one that improves; Astra and Sol decline by 2.2 and 3.6 points.

For repeated requirement changes, test Opus 5.5 against Astra. Prefer Astra if it meets the same acceptance criteria with less total time or spending in your trial.

For bounded edits with strong tests, keep a Flash candidate when it passes your checks at an acceptable total cost. DeepSeek V4.1 Flash’s 55.6% and Gemini 3.8 Flash’s 53.4% do not establish a dependable winner for your repository. Compare retries and review time, not only token prices. The Coding Agent Index cost-per-task guide covers that separate purchasing decision.

Keep checkpoints distinct. The live board separately lists DeepSeek V4.1 Flash 1017; the table above uses the earlier DeepSeek row. Consult the DeepSeek Flash version and coding guide before matching a benchmark name to an API model.

What does MaxxBench change?

MaxxBench changes instructions or starting code in 89 DeepSWE tasks while retaining the main engineering problem. unLAP revises the reference solution and tests, including checks for behavior that must remain unchanged.

One task requires separate browser reports. MaxxBench adds lowercase filenames: Chrome.xml must become chrome.xml, while the reporting feature continues to work. The published Testem task contains the full specification.

Another task adds retirement dates to FastAPI routes. The revised requirement asks for UTC timestamps ending in Z, rather than timestamps preserving the supplied time-zone offset.

The reference patches grow by a median of 17 lines, or 3.5%. That describes the additional patch size, not the total work needed to implement each feature. The published runner requires all necessary feature and regression tests to pass.

Why can an agent understand the change and still fail?

unLAP’s trajectory review found agents allowing guesses about hidden tests to override explicit requirements. The reported examples show a mismatch between an agent’s explanation and its submitted code.

In a ytt task, GLM-5.3 recognized that converting uint64 values to float64 could lose precision. It nevertheless assumed sufficiently large values would not be tested. The verifier checked 2^53 + 1, and the run failed.

Across Opus 5, GLM-5.3, and GLM-5.3 Flash, roughly a quarter of failures on mutated checks followed this pattern: understand the change, then revert to the original behavior. Separately, unLAP attributed test-guessing contributions to 16.7% of Opus 5.5’s failures, 30.8% of Opus 5’s, and 40.3% of GLM-5.3’s.

Those percentages describe the researchers’ classified failures, not all requests or your production failure rate. The trajectory analysis supports inspecting the final behavior rather than accepting a correct explanation as proof.

Which access route should you use for a trial?

Use an existing eligible account first when it already exposes your candidate. For the Opus trial, these are the access routes as of 2026-11-16.

RouteRequirements and billingChoose it when
Claude Code subscriptionYour claude.ai account and a supported paid plan. Confirm the candidate is available and you have usage remaining. Web subscriptions require a credit or debit card.You already use the tool and want to evaluate that complete setup.
Anthropic APIA Console account, API credentials, and funded billing. Most organizations use prepaid credits. Input, output, and cache tokens have separate billing rules. Credit/debit cards are accepted.You want direct vendor API access and separately measured usage.
BetterTokenYour own account, API Key, and balance, with model-specific usage charges. Check payment methods in the console and use a documented tool integration.You want pay-as-you-go access to listed models through a separate API provider.

Check Claude’s supported locations or Anthropic’s API regions for the official route you choose. Anthropic documents Claude Code authentication and Console billing separately.

Try Opus 5.5 through BetterToken in Claude Code

As of 2026-11-16, BetterToken lists claude-opus-5-5 and claude-opus-5. Its Claude Code API access is separate from a Claude.ai subscription. Check current prices before extending your trial.

With Claude Code installed, create your own key in the BetterToken console. The documented Claude Code integration uses https://bettertoken.ai, without /v1.

Run this in Bash:

read -rs BETTERTOKEN_API_KEY && export BETTERTOKEN_API_KEY
export ANTHROPIC_BASE_URL="https://bettertoken.ai"
export ANTHROPIC_AUTH_TOKEN="$BETTERTOKEN_API_KEY"
export ANTHROPIC_MODEL="claude-opus-5-5"
claude -p "Reply with only: OK"

Paste the console key when the first command waits for input, then press Enter. The input is not echoed or stored in shell history.

An OK response confirms the connection. Check the request, model, token usage, and charge in the console. Then run claude in your trial checkout. To compare Opus 5, set ANTHROPIC_MODEL to claude-opus-5 before starting a fresh session.

For 401, check the token and remove a conflicting ANTHROPIC_API_KEY from that shell. In an interactive session, /status should show the BetterToken base URL and ANTHROPIC_AUTH_TOKEN. If the route is wrong, check conflicting settings and restart Claude Code.

Create a BetterToken account.

How do you test one changed requirement locally?

Use a feature your repository already implements, then change one observable requirement. This is a local selection exercise, not a reproduction of MaxxBench.

Save a clean baseline and run the relevant existing tests. Write the new acceptance check before asking the agent to change code. Confirm that the baseline fails the new check for the intended reason.

Give each candidate an identical checkout and a fresh conversation. Keep the tool version, permissions, dependencies, and task instructions fixed. Record the model ID and reasoning setting rather than assuming max, xhigh, and high represent equal budgets.

For a repository that already produces per-browser reports, use this prompt:

Goal: Make per-browser report filenames lowercase.

Allowed scope: Report filename generation and its focused tests.

Preserve: Report contents, separate files for each browser, and existing filename escaping.

Acceptance: Chrome produces chrome.xml. Firefox produces firefox.xml. Existing reporting tests still pass.

Verification: Add focused tests, implement the change, and run the repository's reporting tests. Report the exact test command, its result, and changed files. Stop when these checks pass.

Run the acceptance checks yourself and inspect the generated files. Reject a patch that passes the new check but breaks existing reporting behavior. A correct explanation without the required output is also a failed attempt.

Record failures, retries, elapsed time, inference spending, and review time. Divide total inference spending, including retries, by the number of changes that pass all acceptance checks. Track human review time separately.

A single successful filename change is a smoke test. Before making a team-wide choice, repeat the process on representative changes from your own backlog. Keep a lower-cost candidate only while it meets those acceptance criteria; escalate the task when it does not.

What should you not infer from the scores?

The score gaps are evidence for further adaptation testing, not a measured percentage of memorization. Datacurve’s DeepSWE release contains 113 tasks; MaxxBench modifies 89. unLAP also revised reference solutions and tests. The published before-and-after columns therefore do not isolate requirement changes as the only difference.

These are results reported by the benchmark’s creators, with different reasoning settings across models. Do not interpret a small lead as statistically decisive without repeated, matched trials. Nor should Opus 5.5’s 2.2-point increase be read as proof that changed requirements make it more capable.

Finally, your local Claude Code trial evaluates a different agent setup from the published mini-swe-agent runner. Use its outcome to choose a tool-and-model configuration for your repository, not to claim you reproduced the leaderboard.

Ready to optimize your LLM workflow?

Join thousands of developers building faster, smarter, and more cost-effective AI applications with BetterToken.

Get Started for Free