AI Cost of JavaScript Frameworks: Benchmark and Self-Test
Hono+htmx had the lowest median quoted API cost in Nolan Lawson's 12-framework benchmark, while Angular had the highest. Compare cost, time, tokens, and source lines; separate observed tooling behavior from causal claims; and run a small Claude Code experiment that keeps cache usage, failed attempts, and acceptance checks visible.
Contents
For a content-heavy app, shortlist Hono+htmx and Astro when AI coding cost matters. In Nolan Lawson’s benchmark, Hono+htmx had the lowest median quoted API cost: $2.67, against Angular’s $4.97. For an existing project, test a representative change before choosing a different framework.
Which framework had the lowest AI cost?
Use these results to choose trial candidates for a content-heavy application. Lawson published the experiment on October 28, 2026. The published runs and measurements used Claude Code 2.1.310 with claude-opus-5-5 at high effort.
Each agent built the same PokéAPI application with search, filters, and detail pages. The table combines the published results as of 2026-11-06. Versions identify the tested framework packages, rather than releases recommended for installation today.
| Framework and tested version | Quoted API cost, USD median (min–max) | Minutes | Tokens, millions | Source lines |
|---|---|---|---|---|
| Hono 4.13.14 + htmx 2.0.11 | $2.67 ($2.23–$3.19) | 8.6 | 4.26 | 1,370 |
| Astro 7.4.2 | $2.76 ($2.16–$3.63) | 8.2 | 4.13 | 1,461 |
| Nuxt 4.6.2 | $2.82 ($2.39–$4.25) | 9.0 | 3.85 | 1,415 |
| Vanilla with Vite 8.3.5 | $2.98 ($2.24–$3.37) | 10.1 | 5.00 | 1,492 |
| Next.js 16.4.3 | $3.14 ($2.58–$4.13) | 10.9 | 4.99 | 1,539 |
| Qwik 1.20.2 / 1.20.3 | $3.27 ($2.35–$3.80) | 10.7 | 5.64 | 1,744 |
| TanStack Start 1.169.4 | $3.54 ($3.11–$3.96) | 10.3 | 7.07 | 1,697 |
| SolidStart 2.1.0 | $3.79 ($2.84–$4.81) | 11.2 | 7.13 | 1,630 |
| React Router 8.5.0 | $3.81 ($2.82–$3.97) | 12.5 | 7.58 | 1,564 |
| SvelteKit 3.1.0 | $4.28 ($3.59–$4.65) | 10.8 | 8.53 | 1,611 |
| Gracile 0.11.0 + Lit 3.3.3 | $4.65 ($3.92–$5.07) | 12.1 | 9.04 | 1,717 |
| Angular 22.2.3 | $4.97 ($3.19–$6.82) | 17.2 | 9.76 | 2,009 |
Each metric is an independent median. Cost, time, and tokens include five attempts per framework. Angular’s code size uses three apps because two could not be measured behind the reverse proxy.
Tokens include uncached input, cache writes, cache reads, and output across the session. Source lines exclude blanks, JSON, lockfiles, tests, generated files, and copied design files; comments count.
These dollars are Claude Code’s quoted API costs. Lawson used a subscription, so the table does not describe his actual per-build payments. It also excludes your hosting bill and human review time.
Using the rounded medians, Angular’s gap is $4.97 − $2.67 = $2.30, or 4.97 / 2.67 ≈ 1.86 times Hono+htmx’s cost. That is about 86% higher for this task.
The Next.js versus Astro difference is $0.38 per build, with overlapping observed ranges. Hono+htmx and Astro differ by only $0.09. Those gaps justify a small trial, rather than a confident universal ranking.
Why were SvelteKit and Angular more expensive?
Extra documentation reads and larger generated applications are plausible contributors, but the experiment does not isolate their effects.
Lawson observed the SvelteKit agent reading the v3 migration guide in every run. He suggested that v3 might be newer than the model’s training data. The official migration guide documents real changes, including moving configuration into Vite. It does not establish what the model learned during training.
For Gracile, Lawson reports repository cloning or inspection of compiled files in node_modules. That is additional research work. The benchmark does not measure how much of the cost gap came from those reads.
Angular produced a median 2,009 source lines, versus Hono+htmx’s 1,370. The latter is about 68% of the former, consistent with Lawson’s rough 70% comparison. Larger files can increase future reading and editing work, but this experiment did not measure maintenance costs.
Token totals also need care. Nuxt had the lowest token median, 3.85 million, but did not have the lowest dollar median. Cache usage changes the relationship between total tokens and price. Keep the token categories separate when interpreting cost.
Do framework MCP servers and skills help?
This benchmark does not establish whether MCP servers or skills reduce cost. It records uneven use of available tools, without a paired on/off comparison for each framework.
Lawson installed official tooling where available. Several frameworks had none. His observations included:
| Framework | Observed use across five runs |
|---|---|
| Hono+htmx | Hono Skill tool used in 5/5 |
| SvelteKit | MCP used in 5/5 |
| Nuxt | MCP used in 0/5 |
| Next.js | Skill files read in 1/5; AGENTS.md read in 2/5 |
| TanStack Start | Bundled skill files discovered in 2/5 |
Nuxt’s unused MCP does not prove that it is ineffective. SvelteKit’s higher cost does not prove that its MCP caused the difference. Installation, invocation, and an improvement in completed-task cost are separate questions.
Next.js’s current guidance separates version-matched documentation from skills that coordinate work. Record which resource the agent actually used instead of treating every documentation read as an MCP call.
For your own test, check server availability with claude mcp list or /mcp, then inspect actual tool calls. Compare one framework with tools absent, available, and explicitly requested. The MCP context measurement guide provides a focused follow-up.
What does the experiment leave unresolved?
Treat the results as a screening test for small content-heavy applications. Four limits should change how you use them:
- Workload: a catalog with modest interaction does not represent a complex editor, dashboard, or game. Rendering architecture was also left to the agents.
- Sample size: five attempts produce useful examples and ranges, but close rankings remain uncertain. The displayed ranges are observed minima and maxima, not confidence intervals.
- Environment: one model, agent, and effort setting were tested. Qwik changed from
1.20.2to1.20.3after its first run. The requested Gracile+Lit combination also produced two implementations without Lit. - Outcome: mobile Lighthouse page-load measurements do not measure every interaction or production requirement. Source lines are a size measure, not a demonstrated maintenance-cost forecast.
The Angular exception also matters. The archive records acceptance-test passes for all five attempts, while two production measurements failed. Track test success and successful deployment separately in your own experiment.
How to measure Claude Code cost in your own project
Start with two candidates and three fresh attempts each. Expand to five when results are close. This is a proposed pilot design, not an additional benchmark result.
1. Fix the task and starting state
Compare one bounded feature with identical acceptance checks. For an existing product, use your current framework as one candidate. Build both starting projects before timing an edit experiment; include setup in both if you want to measure creation from scratch.
Put a concrete task in SPEC.md. For example, prepare equivalent catalog pages and the same product fixture, then request:
Add case-insensitive name search to the catalog.
Read products from the existing fixture.
Persist the search query in the URL across reloads.
Show an accessible empty state when nothing matches.
Keep existing routes and the acceptance tests unchanged.Pin dependency versions and retain lockfiles. Install dependencies before an edit trial, and ensure both projects provide npm run build and npm test. Keep an untouched copy of the acceptance tests.
Each trial needs the same source baseline, prompt, model, effort, permissions, intended framework tools, and billing route. Start a new conversation from a fresh copy for every attempt. Alternate framework order and record cache usage.
Also record user instructions, hooks, and plugins that could change behavior. If you prefer the original application and its existing checks, start from Lawson’s benchmark harness.
2. Capture a run and its final result
Use Bash with authenticated Claude Code and jq installed. Check authentication with claude auth status --text; if needed, use claude auth login before testing.
Run in a disposable development container. The allowlist below automatically permits file changes and shell commands. Preapprove intended MCP tool calls using permission rules that load in every trial environment; connecting a server alone is insufficient.
Set a distinct trial_label for each condition. This example uses the documented non-interactive CLI and keeps a unique output directory.
trial_label=astro-01
trial_dir=$(mktemp -d "/tmp/${trial_label}.XXXXXX") || exit 1
claude --version > "$trial_dir/claude-version.txt"
git rev-parse HEAD > "$trial_dir/commit.txt"
trial_status=0
CLAUDE_CODE_EFFORT_LEVEL=high claude -p \
--model claude-opus-5-5 \
--max-budget-usd 10 \
--allowedTools "Read,Edit,Write,Bash" \
--output-format stream-json --verbose \
"Implement SPEC.md without changing dependencies or acceptance tests. Run npm run build and npm test. Report incomplete work." \
> "$trial_dir/session.jsonl" 2> "$trial_dir/stderr.log" \
|| trial_status=$?
printf '%s\n' "$trial_status" > "$trial_dir/exit-code.txt"
jq -s '
[.[] | select(.type == "result")] | last
| if . == null then error("No final result; record an incomplete run")
else {
session_id,
subtype,
is_error,
estimated_usd: .total_cost_usd,
reported_wall_minutes: (.duration_ms / 60000),
modelUsage,
permission_denials
} end
' "$trial_dir/session.jsonl" > "$trial_dir/summary.json"
cat "$trial_dir/summary.json"
printf 'Results: %s\n' "$trial_dir"The $10 threshold is an example budget choice. --max-budget-usd uses estimated API spend and can overshoot. Apply the same threshold to every condition and record budget stops.
As of 2026-11-06, CLAUDE_CODE_EFFORT_LEVEL takes precedence over --effort. Organization effort limits can still cap the selected level. If the model cannot load, resolve the Opus 5.5 configuration before continuing the comparison.
The summary should contain estimated_usd, reported_wall_minutes, modelUsage, and any permission_denials. The JSONL transcript preserves tool events for inspection. Confirm the intended model appears, and record any additional models used.
3. Compare accepted results and retain failures
Use modelUsage for token accounting across the agent and its subagents. Keep inputTokens, outputTokens, cacheReadInputTokens, and cacheCreationInputTokens separate. The result-level usage field excludes subagent activity.
Read the final result once. Do not add cumulative result totals together or reuse a resumed session as a fresh attempt.
Independently run the unchanged tests and production build. Check that dependency and test files stayed unchanged. A successful agent result is not proof that the application meets the spec.
If the final result is missing or malformed, inspect stderr.log and record an incomplete run with unknown cost. Do not enter zero. Resolve permission denials before interpreting an unused MCP as agent behavior.
For each framework, report accepted attempts over total attempts, successful-run medians and ranges, and total quoted cost across all attempts. Calculate batch cost per accepted result as:
Total quoted cost of all attempts / Number of accepted results
If any attempt lacks cost data, label the batch cost incomplete. With no accepted results, report failure rather than a finite unit cost. Record human review time separately.
For manual sessions, current Claude Code cost reporting shows the same kind of estimate in /usage. The commands reference lists /cost as an alias as of 2026-11-06. Subscription allowances and actual provider charges remain separate from that estimate.
Keep an existing stack when a small token saving is outweighed by review effort or migration work. Change that recommendation when the candidate repeatedly passes the same checks with acceptable ownership costs.
After selecting a framework, test another model or access route in a separate experiment. The coding-model comparison can help choose candidates without changing multiple variables in the framework trial.
