Artificial Analysis Coding Agent Index v2.0: Cost per Task
Use Artificial Analysis Coding Agent Index v2.0 to choose a model and harness for your budget. Compare DeepSeek V4.1 Flash, GPT-6.1 Sol, GPT-6 Astra and Claude Opus 5.5, understand the revised benchmarks and API costs that include caching, and measure cost per accepted result in your own repository.
Contents
For the Artificial Analysis Coding Agent Index v2.0, start with Codex + GPT-6.1 Sol (max) for mixed coding work, or test Codex + DeepSeek V4.1 Flash (max) first when API spend is the main constraint. Claude Code + Claude Opus 5.5 (max) is the higher-scoring candidate to test when an accepted change is worth a larger budget. These recommendations follow the benchmark results for different workloads; the dollar figures measure API usage, not subscription prices or guaranteed production costs.
This comparison uses Artificial Analysis’s October 23 launch announcement, with its methodology and live comparison checked as of 2026-10-25. The unit being compared is a model plus its agent harness: the application that manages context, tools, execution and interaction with the model.
Compare the four highlighted configurations
GPT-6.1 Sol offers the most useful starting point for a mixed-workload budget test among these four configurations. DeepSeek lowers the API bill substantially, while Opus raises the headline score; Astra needs a more specific justification.
AA highlighted the following combinations in its launch announcement. All four use the reported max reasoning setting.
| Agent + model | Index v2.0 score | FrontierCode v1.1 score | API cost per task |
|---|---|---|---|
| Codex + DeepSeek V4.1 Flash | 47.3 | 52.6% | $0.39 |
| Codex + GPT-6.1 Sol | 54.1 | 49.4% | $1.50 |
| Codex + GPT-6 Astra | 54.7 | 49.3% | $7.04 |
| Claude Code + Claude Opus 5.5 | 61.7 | 56.0% | $11.42 |
Costs are USD averages from the benchmark workload. The current Claude Code–Codex comparison still lists these four costs as of 2026-10-25, although it rounds displayed scores.
This is a shortlist, not the complete leaderboard or every point on its cost frontier. The live comparison also lists Codex + GPT-6 Luna (max) at $0.16 per task with a rounded index score of 31. DeepSeek is the cheapest of the four highlighted options, not the cheapest measured configuration.
Opus leads the launch ranking. Claude Code + Sonnet 5.5 (max) follows at 59.3, then Muse Code + Muse Wonder Full (max) at 58.9. A high rank alone does not establish the best choice for your workload or spending limit.
Choose by the work you need to finish
Use the relevant component score to decide which pair gets the first trial. The overall index gives three different kinds of work equal importance; your backlog probably does not.
Repeated repository patches: test DeepSeek first
Start with Codex + DeepSeek V4.1 Flash when you have many bounded changes and can check each result with tests and review. Its 52.6% FrontierCode score exceeds Sol’s 49.4% and Astra’s 49.3%, even though its overall index is lower.
That makes a low-cost patch queue a reasonable experiment, not a demonstrated production result. Reverse the choice if rejected patches, extra attempts or reviewer cleanup erase the API savings.
Mixed coding and terminal work: start with Sol
Choose Codex + GPT-6.1 Sol as the initial baseline for work that alternates between changing code and operating a development environment. It adds 6.8 index points over DeepSeek for $1.11 more per benchmark attempt.
The official comparison’s rounded Terminal-Bench 4.0 results are 53% for Sol and 28% for DeepSeek. That gap supports testing Sol first for terminal-heavy work; it does not prove the same gap on your CI repairs or deployment scripts. Choose DeepSeek instead if it reaches your own acceptance threshold at a lower total cost.
Expensive failures: compare Opus with your baseline
Test Claude Code + Opus 5.5 when the cost of an inadequate result outweighs a larger API bill. It has the strongest announced overall score, 61.7, and the highest FrontierCode score among these four, 56.0%.
At $11.42 per benchmark attempt, it costs about 7.61 times Sol’s $1.50. Keep it for a task category only if your comparison shows enough improvement in accepted changes, review time or retries to justify that premium.
Considering Astra: require a task-specific advantage
Do not upgrade from Sol to Astra solely for a 54.1-to-54.7 index increase. The published cost rises from $1.50 to $7.04: about 4.69 times the spend for 0.6 additional index points.
Astra can still win on a particular task distribution. For example, its rounded Terminal-Bench score is 56%, versus Sol’s 53%. Start with Sol and promote Astra only after observing an advantage that matters in your own work; the small aggregate gap alone does not establish one.
Calculate a budget without turning scores into success rates
Multiply the reported cost by attempts, and treat the result as a workload-dependent estimate. Do not multiply a task count by the index percentage and call the result completed fixes.
For an illustrative 1,000 attempts with AA’s task mix and unchanged average token costs:
| Configuration | Estimated API spend for 1,000 attempts |
|---|---|
| Codex + DeepSeek V4.1 Flash | $390 |
| Codex + GPT-6.1 Sol | $1,500 |
| Codex + GPT-6 Astra | $7,040 |
| Claude Code + Opus 5.5 | $11,420 |
The calculation is simply attempts × reported mean API cost per attempt. These estimates exclude your infrastructure, paid external tools and human review. They are not estimates for 1,000 accepted pull requests.
To reproduce a cost-per-index-point calculation, use the index on its published 0–100 scale:
- DeepSeek:
$0.39 / 47.3 ≈ $0.00825per index point. - Sol:
$1.50 / 54.1 ≈ $0.02773per index point. - Astra:
$7.04 / 54.7 ≈ $0.12870per index point. - Opus:
$11.42 / 61.7 ≈ $0.18509per index point.
This ratio describes the snapshot; it is not cost per successful task. The index combines different scoring systems, and its benchmark weights differ from the cost metric’s pooling. A low ratio can also favor a cheap configuration whose absolute performance is below your minimum acceptable level.
For a purchasing decision, measure total spending divided by accepted results on your own task set. Include spending on failed attempts and escalations in the numerator.
What changed in Coding Agent Index v2.0?
Version 2.0 replaces the software-engineering component and changes repository-Q&A grading. Terminal-Bench 4.0 remains unchanged from index v1.5.
The published index methodology specifies:
| Component | Tasks | What it measures | Scoring |
|---|---|---|---|
| FrontierCode v1.1 Main | 100 | Changes to real repositories | Zero if a blocking check fails; otherwise 70–100% |
| Terminal-Bench 4.0 | 66 | Tasks completed through a terminal | Test-suite pass/fail |
| SWE-Atlas-QnA-AA | 124 | Questions about repository behavior | A task passes only if every rubric criterion passes |
That is 290 distinct tasks. AA runs three attempts per task and averages them; it does not pick the best of three. It then averages the three benchmark scores with equal weight.
Consequently, the 124-question component does not outweigh the 66-task terminal component. And an index score of 61.7 does not mean that 61.7% of ordinary coding tickets will be completed.
FrontierCode rewards changes that meet maintainer standards
FrontierCode replaces DeepSWE v1.1 and makes code quality part of the evaluation. Cognition’s benchmark design considers whether a change meets a repository maintainer’s requirements, including scope and code quality, rather than checking functional correctness alone.
AA uses the private 100-task Main set. Failing any blocking requirement yields zero; passing every blocker unlocks a 70–100% score based on the remaining checks.
Internet access remains available. Under FrontierCode 1.1’s fair-use rules, reading documentation is allowed, while retrieving solution material can invalidate a run. AA applies a modified version of Cognition’s scanner and assigns flagged attempts zero.
The privacy of the task set reduces one source of contamination; it does not mean an upstream solution cannot exist online.
SWE-Atlas-QnA-AA changes the judge and rubric wording
The repository questions stay the same, but their grading changes. AA retains 124 tasks and 1,306 criteria, rewords 127 criteria, and uses GPT-6 Luna at xhigh with five votes per criterion.
The published dataset and judge configuration specify majority voting for each criterion. Task success still requires passing every criterion, so a technically plausible answer that misses one required detail can fail.
AA describes the revisions as making verdicts more consistent. The important consequence for comparisons is that the resulting score belongs to this revised grader, not the old rubric-and-judge combination.
Why lower scores do not prove a model regression
The drop between index versions reflects a changed evaluation, so it cannot be read as a deterioration in the same models. Compare configurations within v2.0 before comparing their numerical scores with an older index.
In its official announcement thread, AA reports that the mean index for 21 shared configurations fell from 54.8 to 47.4, with 20 of 21 declining. The same configurations averaged 65.6% on DeepSWE v1.1 and 45.1% on its replacement, FrontierCode v1.1.
Repository-Q&A scores also fell on average, from 58.8% to 57.1%; 14 of the 21 configurations moved down. Terminal-Bench was unchanged. With equal benchmark weights, those reported component changes explain the 7.4-point overall decline:
((45.1 - 65.6) + (57.1 - 58.8) + 0) / 3 = -7.4
This is a different measuring scale. It also changes relative results: stricter grading does not reduce every configuration’s score by the same amount.
What the API cost includes—and what it leaves out
AA’s cost metric models token billing, including caching where supported. It does not price a Claude or ChatGPT subscription, or the full engineering work needed to ship a change.
The cost and token-usage sections distinguish uncached input, cached input, cache writes and output. The public page also exposes token-distribution and cache-hit-rate views. Treating every input token as a full-price cache miss would misrepresent this calculation.
Cost is pooled across task attempts, while the headline index equally weights the three benchmarks. AA excludes missing telemetry from an efficiency average instead of substituting zero. Those choices are another reason not to treat cost divided by index score as cost per resolved issue.
Your costs can change with context size, retries, reasoning effort and provider cache behavior. A different endpoint or harness can change the comparison even when its model label looks familiar.
Also keep AA’s standalone FrontierCode model evaluation separate: it uses AA’s Stirrup harness. Its model scores are not interchangeable with the Claude Code and Codex configuration scores above.
Run a small comparison in your repository
Test two configurations against the same acceptance criteria before routing a whole backlog to either one. A small pilot should produce a decision about your task category, not a claim that you reproduced AA’s private benchmark.
- Select representative work. Start with ten real tasks from the category you intend to automate. Write the expected behavior and acceptable scope before running either agent.
- Hold the starting conditions steady. Use the same repository commit, task prompt, available tools and permissions. Record the CLI version, exact model ID or revision, reasoning setting and provider route.
- Run each configuration independently. Use a fresh checkout and session for each attempt. Record failed attempts and additional tries instead of retaining only the best-looking patch.
- Evaluate the usable result. For patches, run the relevant tests and review correctness, scope and maintainability. For repository questions, verify the explanation against the code and any required execution evidence.
- Compare accepted outcomes and full costs. Record API charges, paid tool usage and runtime costs, and value review time at your team’s chosen hourly rate. Divide total expenditure by accepted results; record subscription fees and overages separately if evaluating a subscription route.
Model identity deserves particular care with DeepSeek. As of 2026-10-25, its native API documentation names deepseek-flash, currently served by DeepSeek-V4.1-Flash-1017. The short leaderboard label alone does not establish that a newly routed request reproduces AA’s evaluated snapshot.
If you use BetterToken for the pilot, follow its separate Codex and Claude Code API integration guides and measure that route’s charges; this is API access, separate from vendor application subscriptions.
Keep the cheaper configuration where it meets your acceptance threshold. Escalate the task categories where the stronger configuration earns back its additional cost through better accepted results or less review work.
