What happens when you benchmark local AI for real agent work?
I tested ten local model lanes on two NVIDIA DGX Spark/GX10 boxes using the kinds of tasks an agent actually touches: tool calls, coding fixes, source-backed summaries, calendar and task proposals, and safety checks. The point is not to crown a universal winner. It is to show how I decide which local model is safe enough to use, which one is merely fast, and which ones should stay out of the main route.
The best default was not the fastest model.
DeepSeek V4 Flash DSpark is the model I would route to first today. Qwen3.6 35B A3B was more than twice as fast, but it missed too many tool-safety cases. If you are building local agents, that tradeoff matters more than a raw tokens-per-second chart.
The results, including the models I would not run.
Cards are ranked by cumulative quality score, then speed. I left the weaker and unfinished rows in because they are useful: they show where speed, safety, coding ability, or operational complexity became the blocker.
Best overall. Primary local model. Best overall current production fit: high direct API/tool scores, full live-agent pass, and 51/75 SWE-bench Verified.
Strong fallback. Strong fallback, but sleeper-injection/tool-safety remains the guardrail reason not to prefer it over DSpark.
Best simple setup. Best simple single-node reference. Quality is real, but concurrency/context limits keep it separate from the dual-node target.
Challenger. Cleaner after disabling thinking, but full tool-eval and knowledge/productivity still trail DeepSeek.
Do not promote. Live-agent pass did not translate to external coding strength; 34/75 SWE-bench keeps it behind current leaders.
Fast, risky. Fastest measured lane by far, but tool-eval safety blocks default promotion. Use only behind validators if revisited.
Older baseline. Kept for provenance because it was easy to confuse with M3. The weak coding score belongs to M2.7, not M3.
Reject. Strong live-agent harness result, but severe SWE-bench miss: only 8/75 resolved with many empty patches.
Do not promote. Tool-eval alone looked deceptively good, but coding and knowledge/productivity gates failed hard.
Planned next. Placeholder row. It stays below evaluated rows until dual-node serve, full pipeline, and SWE-bench rerun finish.
How to read the benchmark.
Most model comparisons collapse everything into one score. I split the work into gates because local agents fail in different ways: a model can write decent code, call tools badly, summarize well, or be too slow to use every day.
API tools
Direct OpenAI-compatible tool-call format, selection, and parallel-call behavior.
API coding
Direct coding prompts. Useful signal, but not a full autonomous-agent benchmark.
Knowledge/productivity
Todoist, Obsidian, calendar, source-backed QA, and privacy-sensitive proposals.
Tool-eval full
External 69-case tool-use suite. Short smoke runs do not replace the full score.
Live-agent
Hermes software-engineer profile editing temp repos and passing tests when run.
External coding
SWE-bench Verified slices/runs normalized to a /100 decision column.
Full ten-model comparison.
Use the table if you want the details behind the cards. Scores are normalized to /100 where I ran the gate. Em dashes mean that part of the benchmark was not run or was excluded from the cumulative score.
| Model / status | Cumulative | API tools | API coding | Knowledge / productivity | Tool-eval full | Live-agent | External coding | Speed | Ops risk | Decision |
|---|---|---|---|---|---|---|---|---|---|---|
| #1 DeepSeek V4 Flash DSpark Best overall Two GX10s · DSpark NVFP4 Stage-C C8 · stable DeepSeek Flash alias preserved GitHub recipe ↗ | 88.8% | 100 | 100 | 79 | 86 | 100 | 68 | 43.8 tok/sec | Medium | Primary local model. Best overall current production fit: high direct API/tool scores, full live-agent pass, and 51/75 SWE-bench Verified. |
| #2 DeepSeek V4 Flash 2x Strong fallback Two GX10 SparkDocker · preserved rollback baseline | 87.3% | 86 | 100 | 84 | 86 | 100 | 68 | 33.4 tok/sec | High | Strong fallback, but sleeper-injection/tool-safety remains the guardrail reason not to prefer it over DSpark. |
| #3 AEON 27B DFlash single-node Best simple setup Single GX10 · 65k/131k context lane GitHub recipe ↗ | 86.4% | 100 | 100 | 95 | 85 | 91 | 48 | 37.4 tok/sec | Low | Best simple single-node reference. Quality is real, but concurrency/context limits keep it separate from the dual-node target. |
| #4 MiniMax M3 W4A16 GPTQ Challenger Two GX10s · M3 W4A16 GPTQ lane | 84.7% | 100 | 100 | 79 | 79 | 100 | 50 | 24.6 tok/sec | High | Cleaner after disabling thinking, but full tool-eval and knowledge/productivity still trail DeepSeek. |
| #5 Hy3 295B NVFP4 MTP1 Do not promote Two GX10s · TP=2/MTP1 challenger lane | 77.6% | 100 | 75 | 68 | 77 | 100 | 45 | 15.1 tok/sec | High | Live-agent pass did not translate to external coding strength; 34/75 SWE-bench keeps it behind current leaders. |
| #6 Qwen3.6 35B A3B NVFP4 Fast, risky Single GX10 · stopped after safety-critical misses GitHub recipe ↗ | 74.9% | 100 | 75 | 89 | 35 | — | — | 95.9 tok/sec | Medium | Fastest measured lane by far, but tool-eval safety blocks default promotion. Use only behind validators if revisited. |
| #7 MiniMax M2.7 NVFP4 Older baseline Two GX10s · older M2.7 lane | 71.1% | 86 | 25 | 58 | 87 | 100 | — | 24.5 tok/sec | High | Kept for provenance because it was easy to confuse with M3. The weak coding score belongs to M2.7, not M3. |
| #8 Nemotron Puzzle 75B A9B NVFP4 Reject Two GX10s · Nemotron Puzzle NVFP4 lane | 65.9% | 71 | 75 | 68 | 70 | 100 | 11 | 20.4 tok/sec | High | Strong live-agent harness result, but severe SWE-bench miss: only 8/75 resolved with many empty patches. |
| #9 Step 3.7 Flash no-MTP Do not promote Two GX10s · stopped lane GitHub recipe ↗ | 45% | 86 | 0 | 5 | 89 | — | — | 19.7 tok/sec | High | Tool-eval alone looked deceptively good, but coding and knowledge/productivity gates failed hard. |
| #10 AEON 27B DFlash dual-node Planned next Two GX10s · TP=2 target 262k context | pending | pending | pending | pending | pending | — | — | pending | Medium | Placeholder row. It stays below evaluated rows until dual-node serve, full pipeline, and SWE-bench rerun finish. |
The rejected rows are the point.
Speed is not safety
The fastest measured row was blocked by full tool-eval safety failures.
Throughput matters for local offload, but not enough to route side-effect-adjacent agent work without validators and safety gates.
Harness names matter
M2.7 and M3 had to be separated visually because their conclusions differ.
The page keeps historical rows explicit so a weak score from an older lane does not accidentally contaminate a newer serving candidate.
Partial rows are still useful
Not-run and planned-next gates remain visible instead of disappearing.
A useful benchmark page should show what has not been tested yet. Pending lanes stay visible so readers can tell the difference between a bad score and missing evidence.
What you can copy for your own local AI tests.
The useful pattern is the promotion system: complete rows, explicit missing gates, safety-aware verdicts, and rollback preservation.
Show every lane you tested. Rejected, older, skipped, and planned rows often explain the decision better than the winner does.
Separate promotion verdict from raw score. A high score with safety failures should still read as “do not promote.”
Publish the parts readers can use: hardware shape, benchmark categories, rough speed, safety misses, and promotion criteria.
Use full-suite results as the decision signal. Short smoke runs are helpful for debugging but too easy to over-trust.
Preserve fallback and planned rows so readers can tell whether a gap is quality, speed, safety, or deployability.
Read this as a field note, not a universal ranking.
These measurements come from my GX10/Hermes setup, so treat them as a field note rather than a universal model ranking. The useful part is the testing pattern: compare quality, tool behavior, coding ability, speed, and operational risk before you make a local model the default for agent work.