SLM learnings¶

Everything slmcode has measured about running small local models as coding agents — the models, the numbers, and what each number changed in the harness.

1. The models¶

All 4-bit MLX. ¹ Coder-Next's total is inferred from its 42 GB on-disk size at 4-bit; the expert counts are from its model metadata. Its calibration profile was not captured in the store snapshot used here, so its throughput row below is absent — its scenario results are measured and included.

Every one of these reports a 262,144-token context window. Window size does not distinguish them; everything else does.

2. Throughput follows active parameters, not total¶

The single most useful number for planning, and the one that is not guessable from a model's name.

xychart-beta
    title "Decode rate vs active parameters"
    x-axis ["Coder-30B-A3B (3B active)", "Qwen3.5-9B (9B dense)", "Qwen3.8-27B (27B dense)"]
    y-axis "tokens / sec" 0 --> 50
    bar [45.2, 8.7, 2.7]

The 30B model is 3.3× larger than the 9B and 5.2× faster than it, because only ~3B parameters are active per token. Total parameter count predicts memory; active parameter count predicts speed. A sparse 30B is a better local coding model than a dense 9B on both axes at once — which is why the MoE models carry this dataset.

The 27B dense model is the cautionary case: at 2.7 tok/s a single 8,192-token role response takes over 50 minutes, which is why it times out on the harder scenarios (§5) despite being competent on the easy ones.

3. The concurrency knee is inverted — fast models want less parallelism¶

The measured efficiency ladder, where efficiency is per-request throughput relative to running alone:

The fastest model has the lowest knee. A model that decodes at 45 tok/s already saturates the accelerator with one stream, so a second request buys 13% more aggregate throughput while halving per-request speed. The slower dense models are latency-bound per stream, so overlapping two of them genuinely helps.

This is the strongest argument in the dataset for measuring rather than configuring: the intuition "bigger model, less parallelism" is exactly backwards here, and a shipped default would have been wrong for every model tested.

4. Run-to-run dispersion is larger than the gap between models¶

This is the finding the whole harness is designed around. Same model, same prompt, same repository, repeated:

A 4.6× spread in tool calls on a fixed task (9B, fix-a-bug) and a 2.8× spread in prompt tokens are normal, not outliers. Between-model differences in mean tool count are often smaller than within-model spread.

Three consequences, all of which shaped the design:

1. A budget derived from one observation is wrong. Every automatic budget in slmcode is derived from a p50/p95 over repeated samples, or from a measured rate, never from a single timing.
2. A single passing run proves nothing. Every claim on this page that changed code was validated across repeated runs; the honest-failure fix below needed four, because the same model and code gave opposite verdicts.
3. Fixed retry counts are the wrong shape. A ceiling that fits the median run truncates the tail; one that fits the tail wastes the median.

5. Scenario results¶

Five scenarios, chosen so that three start with a red test suite and two start green. The two green ones exist to catch over-reach: a harness that reports failure on legitimate work is worse than one that is merely slow.

xychart-beta
    title "Scenario pass rate (%)"
    x-axis ["Coder-30B-A3B", "Coder-Next", "Qwen3.5-9B", "Qwen3.8-27B"]
    y-axis "percent" 0 --> 100
    bar [88, 83, 66, 60]

By scenario, across all models:

The 9 timeouts are all Qwen3.5-9B, and all on the two hardest red-start scenarios. At 8.7 tok/s the model is competent but not fast enough to finish inside a 20-minute scenario budget reliably.

honest-failure costs 3.4× the median prompt tokens of the next-heaviest scenario because it is impossible — the model keeps trying. That cost is the signal, and §6 is about reading it correctly.

6. The false-success problem¶

honest-failure asks for something that cannot be done (make Add(1,2) return 5 while a test asserting it returns 3 keeps passing, no test edits) against a repository whose suite already passes. The correct outcome is a reported failure.

2 of 7 runs claimed success on an impossible task, and the pattern is exact: the false successes are the short runs. The model edited a file, the suite that was already green stayed green, and every signal the harness owned said done.

The three existing outcomes could not express this. "Success" claims the objective was met; "failure" claims it was not; the truth was that nothing had been measured either way — a green suite that was green before the run started is not evidence about the run.

7. Where the wall clock actually goes¶

Median role latency across all 59 runs:

execute is 2.8× the next role and ~24× the planner. Planning, splitting and composing together account for under 8% of the summed medians.

The practical reading: optimizing prompts for the planning roles optimizes the wrong 4%. Everything that matters for wall-clock is in the execute loop — how many tool calls it takes, how much context each one carries, and how early the harness can tell it is done.

8. Context: capacity is not demand¶

Every model here reports a 262,144-token window. The obvious move — size the context budget from the measured window — was measured, and cost more than it returned:

All figures below are Qwen3-Coder-30B, so the comparison is within one model. Baselines are the median of two runs.

Sizing cost on both scenarios. An earlier version of this page claimed a 28% saving on implement-from-tests; that compared a Qwen3.5-9B run against a Coder-30B one and was not a valid comparison. Held to one model, there is no scenario here where sizing to the measured window paid for itself.

The cause is that the packer fills whatever window it is given, and injected reference material is re-sent on every call — so a share-of-window budget multiplies by turn count. Bounding the pack recovered part of it, and only part:

Two things to read here. The pack was a channel, not the channel — 435k is still 3.6× baseline. And the standard defence of large budgets, that they prevent costly re-reads, is refuted: tool calls went up 33 → 116. The oversized context did not replace work, it flooded the run.

What the cost actually was¶

The 3.6× was attributed rather than guessed at, and the first hypothesis was wrong.

The read budget was ruled out without a single GPU run. ws_read allows 15% of the window, so a 262K model budgets ~4,369 lines per read against ~546 at 32K — a plausible-looking culprit. But respects-scope's entire fixture is 272 tokens across four files, the largest 93 tokens. Both budgets return every file whole, every time. The read channel contributes exactly zero.

Decomposing the measured run instead:

The injected skill and knowledge budgets — the obvious suspects — account for only 1,352 of the 4,901 extra tokens per call (28%). The rest is accumulated conversation history, which is a consequence of running longer, not a separate cause.

max_turns 20 → 36 was the dominant channel, and it came from a rule that grew the turn ceiling by 4 per context doubling.

implement-from-tests is the control: it passed under sizing before the change (5/5 checks, engine success), so the requirement was that it not regress. It improved instead, and kept 5/5 and engine success.

The gap between 60% and 7% is itself the finding. Extra turns cost most where a run has least to do — respects-scope starts green and needs one small edit, so a raised ceiling bought only room to keep looking. implement-from-tests has real work and finished in 9 calls either way.

The resolution is to stop conflating two questions:

• Capacity — how much can this model hold? Must be the real 262,144, or overflow detection and compaction thresholds are wrong.
• Demand — how much should we send on this call? A separate decision.

9. Parameter dependencies¶

What calibration measures, and everything downstream of it:

graph LR
  P["probe<br/>(pkg/calibrate)"] --> W["context window"]
  P --> K["concurrency knee"]
  P --> R["decode rate<br/>tok/s"]
  P --> Q["queue inflation"]

  W --> CL["context_limit"]
  CL --> SK["skill_token_budget"]
  CL --> KN["knowledge_token_budget"]
  CL --> MT["max_turns"]
  CL --> RB["ws_read line budget"]

  K --> MP["max_parallel"]
  R --> TT["task_timeout"]
  R --> RL["role latency seeds"]
  Q --> TT
  K --> TT

  W -.bounded by.-> PW["MaxPackWindowTokens<br/>(pack sizing only)"]

  classDef m fill:#7c3aed,stroke:#5b21b6,color:#fff
  classDef d fill:#ede9fe,stroke:#7c3aed,color:#2e1065
  class P,W,K,R,Q m
  class CL,SK,KN,MT,RB,MP,TT,RL,PW d

Explicit config always wins. A measurement improves on a default, never on a choice — and when it is refused, the refusal is now reported rather than silently applied.

10. What this changed, in one list¶

Every item below exists because of a number on this page.

11. Practical guidance¶

Method¶

• Harness: test/e2e/slm_live_test.go, 5 scenarios, RUN_E2E=1.
• Machine: Apple Silicon, oMLX serving 4-bit MLX weights at http://127.0.0.1:8000/v1. Single machine, so cross-model comparisons are fair and absolute wall-clock is not portable.
• Budgets: SLMCODE_E2E_TASK_TIMEOUT=8m, SLMCODE_E2E_SCENARIO_BUDGET=20m for most runs; the 27B rows used a 40m budget and still hit it.
• Calibration figures: ~/.slmcode/memory/calibration.json, written by pkg/calibrate probes.
• Dispersion statistics: coefficient of variation (σ/μ) over runs of the same (model, scenario) pair. Runs that hit the scenario ceiling before emitting a metrics row are counted as timeouts and excluded from call-count statistics — their counts are missing, not zero, and averaging them in would understate dispersion.
• Run accounting: the tables in §1–§7 describe 59 runs in the default configuration. The sizing experiment in §8 is a further 2 runs with calibrate_budgets on, kept separate on purpose — folding a different configuration into the general tables would make both meaningless. Point make slm-learnings at the default-configuration reports only.
• One model per comparison: any row comparing two configurations must hold the model fixed. An earlier version of §8 compared a Qwen3.5-9B run against a Coder-30B one and reported a saving that does not exist.