Seven coding models, two real repositories, one frozen harness

Yesterday I published a comparison of seven cheap coding models working on one ten-minute frontend task. Its conclusion was too broad for what it measured. One small task, one repository and mixed client harnesses can describe those particular model and client combinations. They cannot rank models.

So I rebuilt the test around two longer production jobs, one harness and an evaluator I froze before any model started. The headline answer changed. Luna became the sensible default rather than the expensive option, MiniMax wrote the best heavy refactor, and yesterday's value pick fell apart.

Two products, two unrelated domains

ViewRight is my local-first document viewer and editor for Windows and macOS, built with Tauri 2 and Rust around a React 19, TypeScript and Vite 8 interface, with SQLite for local state and PDF.js for rendering. Its task replaced whole-file PDF buffering with range streaming, built one bounded scan-once search index per document and suspended page and thumbnail rendering when a tab went inactive. Doing that touches loading, cancellation, concurrency, React lifecycle, canvas cleanup and viewer state at once. The stop-loss was 60 minutes, 16 files and 1,600 changed lines.

SellRight is my multi-tenant ecommerce platform for operators running several brands, a REST-first, Postgres-backed alternative to standing up a separate Shopify, Vendure or WooCommerce-style backend per brand. It runs on TypeScript, Hono with Zod and OpenAPI, Drizzle and Postgres with row-level security, a Qwik storefront and a React and Vite admin, and it also powers storefronts, licensing and release delivery for my desktop apps. Its task built release identity, file-backed secrets, fail-closed production configuration, compiled migrations, idempotent instance bootstrap, database-role reconciliation and machine-readable deployment verification. That work runs through configuration import order, secret handling, migration locks, password policy, SQL privileges, runtime scripts and release diagnostics. The stop-loss was 60 minutes, 18 files and 1,900 changed lines.

I will publish the frozen packets, the evaluator, the patches and the full scoring breakdown here and on CodeRight once the review is done.

How the runs were controlled

GPT-5.6 Sol at high reasoning in Codex Desktop ran the experiment. It froze the packets, launched the arms, watched the stop-losses and ran the evaluator. It never edited a candidate worktree or repaired model output.

Every worker went through Codex CLI 0.145.0 into OmniRoute 16.2.12, and every arm got an isolated detached worktree, a workspace-write sandbox, a fresh ephemeral Codex turn per packet segment, the same tool surface and one cumulative clock. ViewRight used three implementation turns plus verification. SellRight used four.

No worker profile set model_reasoning_effort. Labels like Pro and Flash came from the provider routes, not from a Codex intelligence level, so Luna ran at whatever its Codex route defaults to rather than an asserted high setting.

OmniRoute records Luna's requested model as codex/gpt-5.6-luna, its resolved model as gpt-5.6-luna and its provider as codex. Cost stayed out of the score, because the access plans and their receipt coverage are not comparable enough to grade fairly.

The evaluator the models never saw

Workers could write tests, but their tests could not award points. I froze the evaluator before any candidate ran and weighted it: 70 points for hidden requirement behavior, 15 for existing tests, typecheck and syntax gates, 5 for required-file completion, 5 for staying inside the file, line and path bounds, and 5 for diff quality.

Hitting a file, line or wall limit counted as a scored outcome rather than a void run. No accepted arm got a corrective prompt or a source edit from the orchestrator.

Results

Scores compare within a task, not across them. Two unrelated engineering domains do not add up to one combined rank, so I have not invented one.

No arm scored 100, and no accepted arm had its output touched.

What the runs cost, and what I actually paid

At direct pay-as-you-go rates the 14 accepted runs come to about $2.71. My incremental cash charge was $0, because every route stayed inside a subscription or token plan I already hold. That does not make the work free, so where a provider publishes enough detail I also allocated plan capacity.

• GPT-5.6 Luna: $0.0283 for ViewRight and $0.0225 for SellRight at direct rates, both inside my Codex subscription. Strip out every cache discount and they would still be $0.0883 and $0.0905.
• MiniMax M3: $2.37 of that $2.71 total. ViewRight used roughly $0.13 to $0.15 of MiniMax Token Plan capacity and SellRight roughly $0.16 to $0.20, depending on which tier you allocate against.
• DeepSeek V4 Pro: $0.0277 and $0.0987, both through ClinePass. ClinePass publishes no per-model quota share, so $0 incremental is the only figure I can defend.
• OpenCode Go routes: DeepSeek V4 Flash, both MiMo variants and Tencent Hy3. At the regular $10 plan price their allowance allocations ran from under $0.0002 to about $0.0143.

Those figures come from unique successful-call receipts, pricing uncached input at the normal input rate, cached input at the cached rate and total output at the output rate. Reasoning tokens already sit inside the output totals, so I did not charge them twice. No Luna request crossed OpenAI's 272K-input surcharge threshold.

Which model to buy depends on which plans you hold

The $0 incremental charge is the part worth arguing about. Ranking these models by API price answers a question most people are not asking, because the marginal cost of a run is usually set by a subscription that is already paid for.

If you have a Codex subscription, Luna is the default and the comparison is over. It never placed worse than second, finished both jobs fastest, and costs nothing extra.

If you have a MiniMax token plan, M3 is the rational pick for heavy refactor work. The best ViewRight implementation in this test cost about $0.15 of plan capacity, which is a bargain for the top score. Its 42 on SellRight is the warning: buy it for the job it won, not as a general-purpose worker.

If you are paying per token with no plans, DeepSeek is the value case. V4 Flash scored 62 against Luna's 69 on SellRight for roughly a tenth of Luna's cost, and V4 Pro landed within one point of Luna on ViewRight for about 2.8 cents. Both took two to four times longer, which matters only if someone is waiting. Flash also hit the file ceiling, so budget for review time rather than assuming the cheap run is finished.

Pricing sources: OpenAI, DeepSeek, Xiaomi MiMo, MiniMax, Tencent TokenHub, OpenCode Go and ClinePass.

What yesterday's post got wrong

The first test ran one CodeRight frontend feature on a roughly ten-minute clock through two coding clients, and scored it by mixing automated gates with my own source review. That measured something real, but it entangled client behavior with model behavior and rewarded models that interpret a request well over models that execute a plan completely.

This one held the harness still. One CLI and one gateway served every worker, two repositories covered unrelated domains, and the packets, stop-losses and evaluator were fixed before launch. Under those conditions most of yesterday's ranking came apart.

Base MiMo, which I had called the best everyday worker, scored 24 and 19. MiMo Pro, which I had said was not worth its premium, beat base MiMo on both jobs. MiniMax led ViewRight outright and then hit SellRight's line ceiling. DeepSeek V4 Pro stayed close to the top of both tables, while Flash did well on SellRight and crossed its file ceiling doing it. Luna finished inside every bound on both jobs, second and fastest on ViewRight, first on SellRight, for under three cents of API-equivalent usage per job.

One finding survived the rerun unchanged. Passing worker-written tests still does not prove the packet was implemented. The hidden behavior checks and source invariants found omissions in runs whose own focused suites were green.

What I would tell someone choosing a worker

Task fit beat any universal ordering. Luna was the most consistent finisher and the best balance of score, time and cost. MiniMax wrote the strongest ViewRight diff but spent 2.5 times Luna's runtime and about 37 times its API-equivalent cost to gain six points. DeepSeek held up across both domains. MiMo's value case, which looked strong on a small task, did not survive a long one.

Two tasks cannot prove that one model is better than another in general. They can show which worker delivered these two frozen plans, through these access paths, inside these limits.

The part I take back to CodeRight is not a preferred worker. It is that this comparison only meant anything because route identity was recorded per request, every worker was bounded, and nothing graded its own output. CodeRight's write-up of the harness goes into that, as does why I treat the runtime as the product.