Does the code compile? Six local models on a DGX Spark
Most "DGX Spark benchmark" posts measure tokens per second. That answers how fast, not which model to install. For a developer the second question reduces to something a machine can check: does the code compile, and do the tests pass?
Setup
- ASUS Ascent GX10 (NVIDIA GB10, 128 GB unified memory)
- Ollama for models that fit side by side,
llama-server(llama.cpp) for the rest - Harness: llm-compile-bench: one prompt per
language with a fixed signature, hidden tests,
cargo/g++/pytest/go test/tscas judges
Models
| Model | Quant | Size | Runtime |
|---|---|---|---|
| Nemotron 3.5 Lightning 30B-A3B | IQ4_NL | 17 GB | Ollama |
| Qwen3.8-27B | Q4_K_M | 18 GB | Ollama |
| gpt-oss-120b | MXFP4 | 65 GB | Ollama |
| Qwen3.6-35B-A3B ("qwopus36") | Q6 + MTP | ~30 GB | llama.cpp |
| DeepSeek V4 Flash 0731 | UD-IQ2_XXS | 91 GB | llama.cpp |
| Qwen3-Coder-Next 80B-A3B (from 2 November) | Q6_K | 62 GB | llama.cpp |
Results
Six tasks, five runs per model (30 attempts each), temperature 0.2, 32k completion tokens. "Wall-to-pass" is the average time over the attempts that passed: the number that matters when you are waiting for a usable answer. "Flaky" lists tasks that passed in some runs and failed in others.
| Model | Compiles | Tests pass | Wall median | Wall min..max | Wall-to-pass | tok/s | Flaky tasks |
|---|---|---|---|---|---|---|---|
| Qwen3.8-27B | 30/30 | 28/30 | 54 s | 15..134 s | 57 s | 31 | go_worker_pool 4/5, rust_iso8601 4/5 |
| gpt-oss-120b | 30/30 | 26/30 | 22 s | 10..63 s | 27 s | 39 | go_worker_pool 1/5 |
| DeepSeek V4 Flash | 26/30 | 25/30 | 118 s | 55..1638 s | 169 s | 20 | go_worker_pool 3/5, react_counter 4/5, rust_iso8601 3/5 |
| Qwen3.6-35B-A3B | 18/30 | 17/30 | 22 s | 11..326 s | 24 s | 95 | rust_iso8601 2/5 |
| Nemotron 3.5 Lightning | 18/30 | 15/30 | 42 s | 13..233 s | 34 s | 86 | cpp_lru 2/5, go_worker_pool 2/5, ts_debounce 1/5 |
The first single run told a different story: gpt-oss-120b went 6 for 6 and looked like the clear winner. Over five runs its Go worker pool passed once in five. Qwen3.8-27B, second on that first run, is the most reliable model here. The Go task is the one that separates the field: every model is flaky on it, and it is the only task about concurrency.
Update, 4 November: five runs of 36 tasks, six models
Five full runs of the 36-task set, 180 attempts per model, same box and settings: temperature 0.2, 32k completion tokens. A sixth model joined: Qwen3-Coder-Next, a mixture-of-experts model with 80B parameters, 3B of them active, trained for code.
| Model | Compiles | Tests pass | Wall median | Wall-to-pass | tok/s |
|---|---|---|---|---|---|
| Qwen3.8-27B | 162/180 | 151/180 | 65 s | 76 s | 31 |
| gpt-oss-120b | 162/180 | 149/180 | 25 s | 26 s | 41 |
| DeepSeek V4 Flash | 124/180 | 118/180 | 175 s | 192 s | 16 |
| Qwen3-Coder-Next | 133/180 | 101/180 | 9 s | 9 s | 49 |
| Qwen3.6-35B-A3B | 118/180 | 100/180 | 29 s | 29 s | 95 |
| Nemotron 3.5 Lightning | 97/180 | 79/180 | 74 s | 63 s | 88 |
Tests passed by language, out of 30 (six tasks, five runs):
| Model | C++ | Go | Python | React | Rust | TypeScript |
|---|---|---|---|---|---|---|
| Qwen3.8-27B | 26 | 26 | 25 | 28 | 21 | 25 |
| gpt-oss-120b | 27 | 24 | 26 | 23 | 24 | 25 |
| DeepSeek V4 Flash | 30 | 21 | 21 | 20 | 16 | 10 |
| Qwen3-Coder-Next | 23 | 11 | 21 | 20 | 10 | 16 |
| Qwen3.6-35B-A3B | 20 | 12 | 21 | 22 | 11 | 14 |
| Nemotron 3.5 Lightning | 25 | 6 | 22 | 14 | 6 | 6 |
What the five runs say:
- The top is a tie. The single run of 30 October put gpt-oss-120b ahead; five runs put Qwen3.8-27B two attempts ahead out of 180. That is noise. It is the same reversal as on the six-task set: report the spread.
- gpt-oss-120b is still the one to keep loaded. Same pass rate, three times faster to a passing answer, and the best Rust in the field (24 of 30). Qwen3.8-27B is the most even across languages.
- A coder-trained model did not rescue the 3B-active class. Qwen3-Coder-Next passed 101, the general Qwen3.6-35B-A3B 100, and both stay weak in Go (11 and 12 of 30) and Rust (10 and 11). After one run I thought the coder model was better at Rust; five runs say it is not. What it does have is time to an answer: 9 s, because it answers without reasoning first.
- DeepSeek V4 Flash is a lower bound, and part of the gap is mine. In run 1 its server was not reachable for 19 attempts: that is the box, not the model. Without them it passed 118 of 161. A mistake in my local config left its request cap at 1800 s instead of 2700 s, which cost 9 more attempts, and it ran out of the 32k token budget 19 times. Its C++ was perfect, 30 of 30; its TypeScript was the worst in the field, 10 of 30.
- Tokens per second still ranks the field backwards at the ends. The two fastest generators, at 95 and 88 tok/s, finish fifth and last.
Raw CSVs and the per-task flaky list are in the repository under results/gb10.
Update, 30 October: 36 tasks
The task set grew from 6 to 36: six per language instead of one. The harness learned per-model request timeouts and now records a timed-out request with its real wall time instead of dying; two tasks had their tests fixed (one test contradicted its own prompt, one tested inputs the prompt never promised).
Superseded by the five-run update above; kept for the record. One run of the new set, same box, same settings:
| Model | Compiles | Tests pass | Wall median | Wall min..max | Wall-to-pass | tok/s |
|---|---|---|---|---|---|---|
| gpt-oss-120b | 32/36 | 30/36 | 24 s | 7..59 s | 27 s | 42 |
| Qwen3.8-27B | 30/36 | 28/36 | 69 s | 14..253 s | 66 s | 32 |
| DeepSeek V4 Flash | 29/36 | 27/36 | 208 s | 48..1800 s | 202 s | 15 |
| Qwen3.6-35B-A3B | 23/36 | 20/36 | 35 s | 9..346 s | 26 s | 96 |
| Nemotron 3.5 Lightning | 20/36 | 17/36 | 77 s | 10..378 s | 62 s | 88 |
Three caveats. This is a single run, so by this article's own last takeaway it is a first look, not a measurement; the five-run spread on the new set is next. The numbers are not comparable with the five-run table above, which was the old six-task set. And DeepSeek hit the 30-minute request cap on five tasks, all counted as failures; a run with a longer cap is pending.
What one run does suggest: on the bigger set the order at the top flips, gpt-oss-120b ahead of Qwen3.8-27B, with DeepSeek close behind despite the timeouts. The bottom does not move. Tokens per second still ranks the field backwards: the two fastest generators, at 96 and 88 tok/s, are last.
A first look, before the harness
The same one-line prompt, write a Rust function that parses an ISO-8601 date, sent by hand to four models:
Nemotron 3.5 Lightning 93 tok/s 27 s answered in English, date-only
gpt-oss-120b 44 tok/s 95 s invented "chrono 0.5", missing dep
Qwen3.8-27B 30 tok/s 73 s clean; one missing `use Datelike`
DeepSeek V4 Flash 17 tok/s 659 s best code, 11k tokens of thinking
Capping DeepSeek's reasoning at 2048 tokens made it four times faster, and its answer stopped compiling. Thinking is where its quality comes from.
Takeaways
Tokens per second ranks the models backwards. The two fastest generators, Qwen3.6 at 95 tok/s and Nemotron at 86, pass about half the attempts (56 and 44 percent over five runs of the 36-task set). The slower generators are the reliable ones. If you only benchmark throughput, you install the wrong model.
Qwen3.8-27B is the quality pick, gpt-oss-120b the balance pick. On the 36-task set they are level, 151 and 149 of 180. Qwen3.8 does it at 17 GB on disk, leaving room for a second model next to it, and is the most even across languages. gpt-oss is three times as fast to a passing answer (26 s against 76 s), and it is the one I would leave loaded for daily work.
DeepSeek V4 Flash is the strongest reasoner and the worst daily driver. It solves what the others cannot, at five to ten times the wait, and its run-to-run spread (55 s to 27 minutes on the same prompt) is larger than the gap between the other models. Capping its reasoning makes it fast and wrong.
The small MoE models fail on Rust and Go specifically. Python, TypeScript and React mostly pass; the type system and the concurrency model are where they come apart. Training one for code did not change that.
Half of the first run's failures were mine, not the models'. Missing toolchains on the box (every Rust, Go and TypeScript task would have scored zero), code returned without Markdown fences that the harness refused to read, a token budget set too low for reasoning models. A benchmark harness needs the same scepticism as the models it judges; a preflight check that the judges exist is now the first thing the script does.
One run is not a measurement. The single run had gpt-oss at 6/6 and Qwen 3.8 second. Five runs reversed them. On the 36-task set it happened again. Report the spread, not the mean.