A Local 27B Model Tied Sonnet on Coding. 300 Benchmarks Show Where It Actually Wins.

Every discussion I read about local coding models runs on anecdote. Somebody tried a model for an afternoon, liked it, and posted about it. Somebody else tried it, hated it, and posted about that. Both are honest. Neither is a measurement.

I have a benchmark harness with about 212,000 recorded runs of Claude models on standardized coding tasks. So when Qwen3.8-27B came out in August, I could do something better than form an impression. I could drop it into a corpus that already existed and see where it lands.

It tied Sonnet. 72.29 against 72.32. The confidence intervals have a lot to say about that, and I’ll get to them.

The setup

Qwen3.8-27B is a dense 27 billion parameter model. I ran the 4-bit build through Ollama on an Apple M3 Pro with 36 GB of memory. It sits at 17 GB resident and generates about 6 to 7 tokens per second. No exotic hardware, no cloud fallback.

I used 16 tasks, four each in Python, Go, JavaScript, and C#. I didn’t pick them at random. I ranked all 48 tasks by how well they separate models in my existing corpus, and took the 16 that separate best. Two tasks in the full set return an identical score for every model I have ever run, so measuring those would burn an hour of laptop time and tell me nothing.

Five repeats per task per setting. 80 runs per arm, 160 for the quality experiment, 140 more for the attention experiment.

Scoring is half automated and half judged. The automated half runs the task’s real test suite, a linter, and a complexity check. The judged half is a second model reading the code for readability, structure, and whether it did what the task asked. That second half has a weakness I’ll come back to.

The Claude numbers are not fresh runs. They come from the bare baseline of an existing 480 run cross-language experiment, restricted to the same 16 tasks and scored by the same pipeline. That’s the whole reason this comparison is worth reading. I’m not asking anyone to trust that my two runs happened to be comparable.

Where it lands

Bootstrap confidence intervals, 10,000 resamples, on the composite score. Higher is better.

Paired against each Claude model:

Read that second row carefully, because it’s the one I nearly skipped over. At 80 runs I can’t separate this model from Opus either. My interval is 7.6 points wide. Saying “it tied Sonnet” is true and slightly misleading, because it also tied the model above Sonnet. The honest version is that at this sample size the local model sits somewhere in the same band as both, and I’d need several hundred more runs to place it inside that band.

The Haiku row is the one that changed how I think about model tiers. A small cloud model isn’t a scaled down large one. Haiku scores 29.10 on Go and 19.67 on C# in this set. A local 27B model isn’t competing with the cheap cloud tier at all. It’s competing with the middle one, and beating the cheap tier by 27 points.

Only two comparisons survive per language

This is my favorite table in the whole study, because it’s the one that stops me overselling the result. Eight comparisons, four languages against two Claude models.

Six of the eight are ties. Two are real at this sample size, and they point in opposite directions. The local model beats Opus on JavaScript by 8.16 points. It loses to Opus on C# by 10.98.

The JavaScript interval starts at +0.25, which is close enough to zero that I’d want more runs before betting anything on it. The C# interval is the sturdiest finding in the set, and it happens to be the bad news. Usually is.

If you take one number from this post, take the C# one. Language matters more than model choice here, and it doesn’t move in a direction anyone would guess from the parameter count.

I measured a 10 point penalty for thinking mode. The cause was my own code.

This model ships with thinking mode on. It writes out its reasoning before it answers, and the vendor presents that as a feature. So I ran it as a second arm.

Thinking mode scored 61.82 against 72.29 with thinking off. A 10.5 point penalty, which is enormous. I split the composite into its two halves and got a story that fit beautifully:

The judge liked the reasoning-mode code better by 5.4 points. The tests liked it less by 9.7. So reasoning produces code that reads well and works badly. I had a mechanism, a direction, and a corroborating finding elsewhere in my corpus about self-review making output worse. I was ready to write it up.

Then I looked at which individual runs pulled the average down. Fifteen runs had scored exactly zero. All fifteen were on one task, the largest and hardest in the set. On that task, thinking mode wrote a working file in 1 run out of 15. Thinking off managed 15 out of 15.

Then I checked how many tokens those failed runs produced before stopping. Every one of them stopped at 4,096.

That was the output limit in my own harness, set months earlier for smaller tasks, and not looked at since.

What happened is that the model reasoned for roughly 3,500 tokens, went to write the file, and ran out of budget partway through. My harness recorded a run with no code in it. My scorer gave that a zero, which is the right answer for a run with no code in it.

I raised the limit to 16,384 and ran the same task twice. Both finished. They used 4,986 and 5,185 tokens. I was short by about 900 tokens on a budget of 4,096.

With that one task removed, the two settings are indistinguishable:

Thinking mode does not change code quality on these tasks. It does cost 3,000 to 4,000 extra output tokens per call, and this machine produces about 6 of those per second. So I still turn it off, for the wait and not for the quality.

The bad limit doesn’t bother me much. What bothers me is that afterward I couldn’t tell a run the model failed from a run I truncated. Both look like a zero. So the real fix wasn’t the bigger number. It was a flag that records when a run hits the ceiling, so that ambiguity can’t survive into the next analysis.

The attention test, and why one miss is not a curve

The thing I most wanted to measure was attention, because that’s the known soft spot of open models, and because this one has an architecture that invites the question.

It has 64 layers. Only 16 of them keep a full record of everything they have read. The other 48 use a cheaper method that compresses history as it goes. If you wanted to design a model that loses a detail buried in the middle of a long prompt, that’s roughly how you’d do it. Whether it actually does is a measurement.

So I planted one instruction inside a wall of irrelevant text. Write the exact marker ATTENTION-TOKEN-7F3A as a comment on the first line of every file. Then I searched the output for that literal string. Either it’s there or it isn’t. No judge, no interpretation, and it works identically in all four languages.

I varied prompt length and where in the padding the instruction sat. Padding length stayed fixed within each group, so position was the only thing that moved.

139 of 140. The architecture that looked most likely to fail did not fail.

Now the part I want to be careful about, because it’d be easy to draw a curve through this and I don’t think the data supports one. That 95% is 19 successes out of 20. Its Wilson interval runs from 76% to 99%, which overlaps every 100% cell in the table. One miss out of twenty isn’t a measured attention gap. It’s one miss.

What I will say is narrower and still interesting. The only failure in 140 runs landed in the exact cell theory nominates as weakest: the middle of the longest prompt. Both ends of that same prompt held. If this were noise, it had 139 other places to turn up. Suggestive, then, but not significant. I’d need a few hundred runs at that depth before I’d quote a number.

The failing run is the interesting one anyway. It wrote correct, working JavaScript. It solved the task. It just didn’t write the marker, and nothing in the output hinted that anything had been skipped. A failed test is loud. A dropped instruction is silent, and you find out about it later from something unrelated breaking.

My working rule from this: put constraints at the start or the end of a long prompt. Below roughly 15,000 tokens, position didn’t matter across 60 runs. The advice costs nothing to follow even if the effect turns out to be tiny.

One limit worth stating plainly. I stopped at 55,889 tokens. This model claims a 262,144 token window and the vendor says it stretches to a million. I measured none of that, and nobody should read my 95% as a figure that holds at 200,000 tokens.

The number that reversed my own conclusion

I’d already written this model off for one job, and then a figure in my own logs argued with me.

The job was planning. I write a long brief, the model produces a plan, and I hand the plan to a cloud model to execute. Briefs are long, so I had measured how long a long prompt takes to process before the model emits anything. At 56,000 tokens it took 6.2 minutes, which is fatal for anything interactive.

Then I noticed that inside each group of five repeats, only the first run was slow. Runs two through five took about 45 seconds. I had been reading the average and the average was concealing an enormous split.

The server caches the front of the prompt. If the next request opens with the same text, it reuses that work instead of redoing it. So a long brief costs 6.2 minutes once and 10.5 seconds on every request after that.

My earlier probes never saw this for a dull reason. They never sent the same prompt twice, so they measured the cold case every time and I read the cold case as the normal one. A probe that doesn’t resemble the real usage pattern will mislead you with total confidence.

It reverses the conclusion. A large brief is fine as long as you keep it stable so the cache stays warm. The binding constraint isn’t reading the prompt. It’s writing the answer at about 6 tokens per second, which puts a 1,000 word plan just under three minutes. No cache helps there.

What I cannot claim

Four limits. The first is the one that’d embarrass me most if somebody else spotted it before I said it.

• Roughly half of all runs pass zero tests. That holds for the local model, for Sonnet, and for Opus on these same 16 tasks. The composite still reads 72 because the linter, the complexity score, and the judge carry it. So 72 is a between-model comparison and nothing else. It absolutely doesn’t mean the code works 72% of the time.
• The judge reads code without running it. It gave 81 out of 100 to a body of work where half the tests failed. In one run it gave a perfect score to Go code that didn’t compile. That judge is half of every composite in this post. The comparison stays fair, because the same judge graded the Claude runs the same way, but the absolute number flatters everybody.
• I chose hard tasks deliberately. I selected for separating power, which means I selected the tasks models struggle with. An easier set would hand everyone higher scores and me less information.
• One machine, one model, one quantization. Every figure comes from a 4-bit build on an M3 Pro. I tried the faster Apple-native build and abandoned it, because it climbed to 31 GB under sustained load on a 36 GB machine and locked up the laptop twice.

What I would actually do with this

A local 27B model can take over work you would currently send to the middle cloud tier, as long as you can queue that work. At 6 tokens per second it doesn’t replace anything you sit and wait for. It replaces things you kick off and come back to.

Check your own language before you commit. My only two statistically real results were per language and they pointed in opposite directions. If your codebase is mostly C#, the numbers in this post argue against the swap. If it is mostly JavaScript, they argue for it, on the same model, in the same study.

Turn thinking mode off for coding, and if you leave it on, give the model a much larger output budget than you think it needs. Reasoning spends the same allowance the answer needs.

Make your harness tell you which kind of zero it recorded. My 10.5 point finding was real, reproducible, and had a plausible mechanism I could explain out loud. It was also caused by a limit I typed myself and forgot about. The measurement wasn’t lying to me. My explanation of it was, and a zero looks exactly like a result.

All 300 runs, the harness, the task definitions, and the scoring code are open. I’d much rather argue with somebody about my confidence intervals than trade impressions about which model felt better on a Tuesday.