I Tested Five Local Coding Models — Not One Built a Working App on the First Try
I Tested Five Local Coding Models — Not One Built a Working App on the First Try
Nabil Samari
July 29, 20268 min read
Share
I run a language model on my own machine that I use through Open WebUI for pretty much everything: writing, information retrieval, planning, and sometimes coding. My current model is the 27-billion-parameter Qwen3.6 version. It works surprisingly well as a general model, but coding started hitting a wall.
I gave it the task of building a single-file Python application for DJ booking management. Nothing unreasonably complex. An SQLite database, forms, creating and editing bookings, a search function, validation, and prices stored as whole cents.
The first response cut off mid-stream and contained an indentation error. I raised the model's response limit and gave it another try. The second version was longer but still broken. It had a syntax error, missing methods, deletion via GET request, and prices stored as floats.
I asked the model to audit its own code. It found some of the errors but didn't notice an obvious syntax error or the missing methods. Worse, the model started describing the behaviour of methods that didn't exist in the provided code at all.
At that point I decided to stop tweaking prompts and test whether an actual coder model would help.
Five Models on the Same Machine
The test machine has an RTX 4070 Ti Super with 16 gigabytes of VRAM. System memory is 32 gigabytes. That limits the options pretty effectively: massive flagship models won't fit sensibly, even quantized.
The comparison included my current Qwen3.6 Balanced, Qwen2.5-Coder 14B, Qwen3.6 NEO-CODE, Qwen3-Coder 30B-A3B, and Devstral Small 2 24B.
I built five tests for them. The first handled Finnish currency precisely in cents. The second validated booking data. The third was a constrained bug fix. The fourth audited intentionally broken source code. The fifth required building a complete HTTP and SQLite-based DJ booking application.
The Python produced by the models couldn't just look plausible. I saved the answers to files, ran syntax checks, and executed hidden tests inside a Bubblewrap sandbox. The full application was also launched and its routes tested with real HTTP requests.
The First Result Was Wrong
After the first run, my current general model looked like the winner with 86.53 points. Qwen3-Coder trailed at 78.20. The result seemed clear, but something felt off. Some applications had received good scores despite not actually working.
The fault was in my own test harness.
If an application test crashed partway, the unreachable tests dropped out of the scoring denominator. A broken application could thus get a disproportionately good result simply because not all of its errors were reached. Additionally, the test assumed a specific database column name I hadn't required in the prompt.
I fixed the scoring so that an unreachable test counts as a failed test. I also defined the form fields, status values, and Finnish currency format unambiguously. Then I re-ran all five models from scratch.
This was probably the most educational part of the whole experiment. If the evaluation system is broken, a precise score doesn't make the result any more reliable. A benchmark can look scientific and still lie.
Qwen3-Coder Won the Corrected Comparison
In the corrected test, Qwen3-Coder 30B-A3B took first place with 69.78/100. Qwen3.6 NEO-CODE scored 68.95, Devstral Small 2 got 63.27, the current Balanced model 56.45, and Qwen2.5-Coder 51.11 points.
Qwen3-Coder was also overwhelmingly the fastest. It produced around 152 tokens per second on average. The other large models stayed around 39 to 44 tokens per second.
The speed difference comes from the model's MoE architecture. The model has roughly 30 billion parameters in total, but only about three billion are activated per token. All the weights still need to fit in memory, but less computation is needed.
Qwen3-Coder scored full marks on booking validation and the bug fix. The audit also went well. In price handling, though, it made surprisingly dumb mistakes: a negative value got through and one regex broke due to its own string manipulation.
This Isn't a Universal Leaderboard
The test tells you first and foremost how these models perform on my machine and my tasks. It doesn't prove that Qwen3-Coder is always the best local coding model.
The models were different sizes and quantized differently to fit within 16 gigabytes of VRAM. Qwen2.5-Coder ran as a higher-quality Q6 version, while the 30-billion-parameter Qwen3-Coder ran as a noticeably tighter Q3 version. Only one answer was also taken from each task. My current Balanced model already showed how much the result can vary across two runs: on the first round it handled the pricing task perfectly; on the second it forgot an entire requested function.
The test suite also weights things I personally need. It included Finnish currency format, Python, SQLite, HTTP routes, and error handling. Someone else might test a TypeScript repo, React, or Rust and get a different ordering.
Still, the result gives a usable direction. Qwen3-Coder was strong on small tasks, fast, and fit on my machine. That's why it earned a continuing role, even though a single score isn't any kind of final truth.
Not One Built a Working Application
This was the test's starkest result.
Qwen3-Coder's full application compiled but didn't start successfully. NEO-CODE produced an f-string syntax error. Devstral started its response with a Markdown code fence and didn't close it, so the saved file wasn't valid Python. Qwen2.5-Coder got the server running and the form open, but submitting a booking crashed the handler. The current Balanced model compiled but also didn't survive the startup test.
So out of five models, zero built a working application on a single response.
This doesn't mean the models are useless for coding. They handled constrained tasks far better. Every model scored full marks on the bug fix. Several handled validation perfectly. The problems began when the same response had to pack in architecture, a database, HTTP routes, forms, security, error handling, and startup.
Long code easily looks finished before it is. That's a dangerous combination: the answer is convincing, but the machine doesn't care about convincingness. One wrong bracket is enough.
The Best Coder Model Doesn't Solve the Problem Alone
After the experiment I keep two models on my machine for different jobs.
Qwen3.6 Balanced stays as the general model. It handles conversation, web search, writing, ideation, and other daily work. Qwen3-Coder becomes a separate NiceCode model for coding.
But I'm not planning to use it by asking Open WebUI for a complete application and pasting the answer straight into production either.
The coder model needs access to files and a terminal. It needs to write the code, run py_compile, launch the program, run the tests, read the actual error message, and fix the file. Then the same loop again until the tests pass.
This is where Qwen3-Coder's speed matters more than a single perfect answer. When the model produces 152 tokens per second, you can run several correction cycles without endless waiting.
What I'll Test Next
The next step isn't building a new leaderboard. I want to see what happens when Qwen3-Coder actually gets to work rather than just returning one long answer in a chat window.
I plan to launch it as its own NiceCode model and connect it to a coding agent that can read project files, make bounded changes, and run tests itself. Then the interesting metric isn't whether a perfect application emerges on the first attempt. The interesting question is whether the model finds its own mistake in the test results and gets the program working on the second, third, or fifth cycle.
Generation settings need testing too. In this comparison, Qwen3-Coder used its recommended temperature of 0.7. For precise file work, a lower temperature can reduce unnecessary variation, but pulled too low the model may get stuck on a bad solution. I'll run the same tasks at least at a few different temperatures while keeping the rest of the test locked.
One answer per task also isn't enough to measure reliability. In the next version, each test runs multiple times with different seeds. A model that succeeds once in three tries is a different proposition from one that succeeds every time.
I also want to try varying context size and quantization. The current Qwen3-Coder is a Q3 version to fit within 16 gigabytes of VRAM. A higher-quality Q4 would need partial CPU offload and would be slower, but could still be better if syntax errors drop enough. Devstral deserves another chance in a real agent environment, and new models can be added later as llama.cpp support and reasonably sized quantized versions become available.
Ultimately the most important test happens in a real repo. A small bug, existing tests, and a clear acceptance criterion tell you more about daily usefulness than one giant generation prompt.
Local AI Is at Its Best as a Coworker, Not an Oracle
I like local models because they run on my own machine. Data doesn't go to an external service, usage doesn't cost per token, and you can build the system around your own needs.
But locality doesn't automatically make a model reliable. And "coder" in a model's name doesn't mean the program it writes will actually run.
After this test, my first question is no longer which model writes the most impressive-looking answer. I ask which model works best as part of a system that can verify its work.
Qwen3-Coder won. But the most important part of the whole thing isn't the model.
It's the test that tells you when the model is wrong.
