Does the code compile? Six local models on a DGX Spark

2026-10-27 · Boris Ivanov

Most "DGX Spark benchmark" posts measure tokens per second. That answers how fast, not which model to install. For a developer the second question reduces to something a machine can check: does the code compile, and do the tests pass?

Setup

Models

Model Quant Size Runtime
Nemotron 3.5 Lightning 30B-A3B IQ4_NL 17 GB Ollama
Qwen3.8-27B Q4_K_M 18 GB Ollama
gpt-oss-120b MXFP4 65 GB Ollama
Qwen3.6-35B-A3B ("qwopus36") Q6 + MTP ~30 GB llama.cpp
DeepSeek V4 Flash 0731 UD-IQ2_XXS 91 GB llama.cpp
Qwen3-Coder-Next 80B-A3B (from 2 November) Q6_K 62 GB llama.cpp

Results

Six tasks, five runs per model (30 attempts each), temperature 0.2, 32k completion tokens. "Wall-to-pass" is the average time over the attempts that passed: the number that matters when you are waiting for a usable answer. "Flaky" lists tasks that passed in some runs and failed in others.

Model Compiles Tests pass Wall median Wall min..max Wall-to-pass tok/s Flaky tasks
Qwen3.8-27B 30/30 28/30 54 s 15..134 s 57 s 31 go_worker_pool 4/5, rust_iso8601 4/5
gpt-oss-120b 30/30 26/30 22 s 10..63 s 27 s 39 go_worker_pool 1/5
DeepSeek V4 Flash 26/30 25/30 118 s 55..1638 s 169 s 20 go_worker_pool 3/5, react_counter 4/5, rust_iso8601 3/5
Qwen3.6-35B-A3B 18/30 17/30 22 s 11..326 s 24 s 95 rust_iso8601 2/5
Nemotron 3.5 Lightning 18/30 15/30 42 s 13..233 s 34 s 86 cpp_lru 2/5, go_worker_pool 2/5, ts_debounce 1/5

The first single run told a different story: gpt-oss-120b went 6 for 6 and looked like the clear winner. Over five runs its Go worker pool passed once in five. Qwen3.8-27B, second on that first run, is the most reliable model here. The Go task is the one that separates the field: every model is flaky on it, and it is the only task about concurrency.

Update, 4 November: five runs of 36 tasks, six models

Five full runs of the 36-task set, 180 attempts per model, same box and settings: temperature 0.2, 32k completion tokens. A sixth model joined: Qwen3-Coder-Next, a mixture-of-experts model with 80B parameters, 3B of them active, trained for code.

Model Compiles Tests pass Wall median Wall-to-pass tok/s
Qwen3.8-27B 162/180 151/180 65 s 76 s 31
gpt-oss-120b 162/180 149/180 25 s 26 s 41
DeepSeek V4 Flash 124/180 118/180 175 s 192 s 16
Qwen3-Coder-Next 133/180 101/180 9 s 9 s 49
Qwen3.6-35B-A3B 118/180 100/180 29 s 29 s 95
Nemotron 3.5 Lightning 97/180 79/180 74 s 63 s 88

Tests passed by language, out of 30 (six tasks, five runs):

Model C++ Go Python React Rust TypeScript
Qwen3.8-27B 26 26 25 28 21 25
gpt-oss-120b 27 24 26 23 24 25
DeepSeek V4 Flash 30 21 21 20 16 10
Qwen3-Coder-Next 23 11 21 20 10 16
Qwen3.6-35B-A3B 20 12 21 22 11 14
Nemotron 3.5 Lightning 25 6 22 14 6 6

What the five runs say:

Raw CSVs and the per-task flaky list are in the repository under results/gb10.

Update, 30 October: 36 tasks

The task set grew from 6 to 36: six per language instead of one. The harness learned per-model request timeouts and now records a timed-out request with its real wall time instead of dying; two tasks had their tests fixed (one test contradicted its own prompt, one tested inputs the prompt never promised).

Superseded by the five-run update above; kept for the record. One run of the new set, same box, same settings:

Model Compiles Tests pass Wall median Wall min..max Wall-to-pass tok/s
gpt-oss-120b 32/36 30/36 24 s 7..59 s 27 s 42
Qwen3.8-27B 30/36 28/36 69 s 14..253 s 66 s 32
DeepSeek V4 Flash 29/36 27/36 208 s 48..1800 s 202 s 15
Qwen3.6-35B-A3B 23/36 20/36 35 s 9..346 s 26 s 96
Nemotron 3.5 Lightning 20/36 17/36 77 s 10..378 s 62 s 88

Three caveats. This is a single run, so by this article's own last takeaway it is a first look, not a measurement; the five-run spread on the new set is next. The numbers are not comparable with the five-run table above, which was the old six-task set. And DeepSeek hit the 30-minute request cap on five tasks, all counted as failures; a run with a longer cap is pending.

What one run does suggest: on the bigger set the order at the top flips, gpt-oss-120b ahead of Qwen3.8-27B, with DeepSeek close behind despite the timeouts. The bottom does not move. Tokens per second still ranks the field backwards: the two fastest generators, at 96 and 88 tok/s, are last.

A first look, before the harness

The same one-line prompt, write a Rust function that parses an ISO-8601 date, sent by hand to four models:

Nemotron 3.5 Lightning   93 tok/s    27 s   answered in English, date-only
gpt-oss-120b             44 tok/s    95 s   invented "chrono 0.5", missing dep
Qwen3.8-27B              30 tok/s    73 s   clean; one missing `use Datelike`
DeepSeek V4 Flash        17 tok/s   659 s   best code, 11k tokens of thinking

Capping DeepSeek's reasoning at 2048 tokens made it four times faster, and its answer stopped compiling. Thinking is where its quality comes from.

Takeaways

Tokens per second ranks the models backwards. The two fastest generators, Qwen3.6 at 95 tok/s and Nemotron at 86, pass about half the attempts (56 and 44 percent over five runs of the 36-task set). The slower generators are the reliable ones. If you only benchmark throughput, you install the wrong model.

Qwen3.8-27B is the quality pick, gpt-oss-120b the balance pick. On the 36-task set they are level, 151 and 149 of 180. Qwen3.8 does it at 17 GB on disk, leaving room for a second model next to it, and is the most even across languages. gpt-oss is three times as fast to a passing answer (26 s against 76 s), and it is the one I would leave loaded for daily work.

DeepSeek V4 Flash is the strongest reasoner and the worst daily driver. It solves what the others cannot, at five to ten times the wait, and its run-to-run spread (55 s to 27 minutes on the same prompt) is larger than the gap between the other models. Capping its reasoning makes it fast and wrong.

The small MoE models fail on Rust and Go specifically. Python, TypeScript and React mostly pass; the type system and the concurrency model are where they come apart. Training one for code did not change that.

Half of the first run's failures were mine, not the models'. Missing toolchains on the box (every Rust, Go and TypeScript task would have scored zero), code returned without Markdown fences that the harness refused to read, a token budget set too low for reasoning models. A benchmark harness needs the same scepticism as the models it judges; a preflight check that the judges exist is now the first thing the script does.

One run is not a measurement. The single run had gpt-oss at 6/6 and Qwen 3.8 second. Five runs reversed them. On the 36-task set it happened again. Report the spread, not the mean.