Iron for Local Models

Everything reduces to three numbers

Hardware talk around local models drowns in specs. Almost all of it is noise. Three numbers decide the experience, and once you hold them you can size any machine in your head.

Before the specs, the feeling. The passage below is being written token by token at whatever speed you set. Find the rate where the machine stops being the thing you notice: every spec on this page exists to buy that rate.

They combine into one line worth memorizing:

Instead of reading worked examples, work them. The calculator below is that one line, live. Pick a machine, drag the active-parameter slider, and watch the tok/s move: that motion is the entire hardware story of this document.

Three settings worth trying, the same worked examples the static version listed: desktop DDR5 (about 65 GB/s real) with a dense 8B at q4 lands near 13 tok/s ceiling; the same box with a 3B-active MoE jumps to about 30; a dense 27B collapses to about 4. 128 GB of RAM solves N1 completely and does nothing for N2. RAM capacity buys shelf space, not speed, which is why the whole game on a CPU box is picking models with a small N3. Hold that thought, because the field test below shows the same three numbers flipping their ranking the moment a small GPU appears.

Quantization sets the bytes

Models train in 16-bit weights. Quantization stores them in fewer bits, which shrinks both the footprint (N1) and the bytes touched per token (N3). It is the single biggest lever you control after picking the model.

• q8: 8 bits per weight. Near-lossless. Size roughly equals the parameter count in GB (8B model, about 8 GB).
• q4 (Q4_K_M and friends): 4-something bits. The standard operating point. Size roughly equals params × 0.6 (verified: Qwen3-8B at 5.0 GB, Qwen3-30B-A3B at 18.6 GB). Quality loss is real but small; for tool calling and SQL generation it is rarely the thing that breaks.
• q2-q3: desperation territory. Models get visibly dumber, and tool-call formatting is often the first casualty. Avoid for agent work.

Rule of thumb to keep: q4 size ≈ params × 0.6 GB. It sizes any model you meet on Hugging Face without opening the file list. And quantization applies to the KV cache too, which turned out to matter more than anything else on the field-test box; section 11 has that story.

Two sharpenings from the inference-engineering literature. First, quantization moves BOTH walls at once: fewer bytes per weight is more effective bandwidth for decode, and on GPUs with low-precision tensor cores it also doubles raw compute, which is why one precision step is worth roughly 30-50% in practiceest. Second, not everything tolerates it equally: sensitivity runs weights, then activations, then KV cache, then the attention math itself. Weights shrug off q4; the cache wants the care section 11 gives it; attention stays in full precision in every sane recipe.

Two speeds, and agents stress the ugly one

An inference server does two different jobs. Prefill reads your prompt and builds the model's working state; it is compute-bound and parallel. Decode generates tokens one at a time; it is bandwidth-bound. GPUs chew through prefill so fast nobody thinks about it. CPUs do not.

This matters enormously for agent harnesses, because a harness turn is not a chat message. System prompt plus tool schemas plus history routinely runs 10-20k tokens before your actual question. The field test confirmed the shape of this: the harness front-loaded roughly 20k tokens of MCP tool schemas before the first user wordobs. Drag the prompt slider to harness scale and watch what happens to the first turn on CPU.

The mitigation is prefix caching: the server keeps the computed state of the prompt so the next turn only processes what changed. llama.cpp's server does this per slot. With it, turn one is slow and every later turn is cheap. Without it, every turn is turn one, which the widget above makes painfully visible. When an agent setup on CPU "feels broken," check this before anything else.

One more way to lose that cache: serving stacks unload an idle model after a few minutes (Ollama's default is about five), and the reload throws the prompt cache away with it, so the next turn pays the full reload plus the full prefill again. For an agent box, pin the model in memory: OLLAMA_KEEP_ALIVE=24h. It is the single cheapest lever on perceived speed.

One chart explains both speeds

Every machine has two ceilings: how fast it computes (FLOPS) and how fast memory feeds it (bandwidth). Which ceiling you hit depends on one property of the work: how many operations you perform per byte fetched, its arithmetic intensity. Below a machine's ridge point you are memory-bound and the compute idles; above it you are compute-bound and the bandwidth idles. The book's worked example: an H100 offers 989 teraFLOPS against 3.35 TB/s, so its ridge sits near 295 operations per byteest. Every machine has its own ridge; the shape is universal.

Now place the two jobs on that chart. Prefill fetches each weight once and applies it to the whole prompt at once: thousands of operations per byte, compute side. Decode refetches every weight for every single token, roughly two operations per byte: pinned deep in the memory-bound slope. The formula of section 01 IS that left slope. This is the mechanism behind everything the page has claimed so far.

Watch the mechanism run. Each sweep below is one token being made: every active byte crossing the bus once, past a compute unit that spends most of its time waiting.

The batch slider is the punchline. A server answering two hundred people at once reuses every fetched weight two hundred times, so decode slides right, crosses the ridge, and the GPU finally earns its FLOPS. That is why cloud tokens are cheap and why datacenters batch aggressively. Your agent box runs at batch one, the leftmost, worst point on this chart, which is why memory bandwidth is the entire story for a local box and compute specs barely matter.

The chart also predicts the one near-free lever a batch-one box has: speculative decoding. Since decode leaves most of the compute idle, a small draft model can propose several tokens and the big model verifies them all in a single pass; accepted drafts arrive in bulk. It improves tokens per second but never the first-token wait, and its payoff lives or dies on the acceptance rate, so it gets measured, not assumed. llama.cpp and friends support it via a small draft model from the same family; purpose-built drafts (the EAGLE style) are the current strong default in server engines.

Context is not free

The model's working memory for the conversation, the KV cache, grows linearly with context length and lives in the same RAM as the weights. The field-test box gave this section its concrete anchor: a Qwen3-4B-class model stores roughly 144 KB of cache per token of context at fp16. Slide the context window and watch the cache dwarf the weights.

Two practical consequences. First, budget for it: model at 20 GB does not mean 24 GB of RAM is comfortable; the field's sizing rule is weights plus at least 50% headroom for cache, more for long contextest. Second, set the context length explicitly. Serving defaults are often small and truncate silently, and in an agent harness silent truncation looks like the model forgetting its tools mid-conversation. On a 128 GB box you can afford 32k context with enormous headroom; the point is to configure it, not hope. On a small GPU, the meter above IS the whole game: 64k of fp16 cache is about 9 GB, more than the card itself.

Machines, ranked by the number that matters

Here is the landscape, spec-sheet bandwidth unless noted. The ladder ranks the one number that sets speed; the map after it adds the second axis the ladder cannot show, capacity, which is where the intuition actually lives. Most of these machines are one click away in the calculator of section 01 and the simulator of section 08.

What to run on a 128 GB CPU box

One naming trap first: "Qwen 3.8" is the August 2026 model generation, not "Qwen3, 8B." The open Qwen3.8 models are a dense 27B and a 2.4T-parameter monster; nothing in that generation is CPU-shaped. The CPU-friendly mid-size MoE models live in the earlier Qwen3 / 3.5 / 3.6 lines. All sizes below are 4-bit unless noted.

The pattern to internalize: on CPU, total parameters are nearly free, active parameters are expensive. A 117B MoE outruns a dense 27B by 3-5x while fitting comfortably in your RAM. Every model worth running on this box has "A3B" or "A10B" or "5.1B active" in its spec sheet. One honesty note that cuts the other way: the active-parameter discount is a batch-one property. A server batching many users routes different requests to different experts and lights up nearly the whole model, so MoE stops being cheap at datacenter scale; it stays cheap for exactly the single-user box this page is about. And all of it assumes the CPU is doing the decoding. The moment even a small GPU enters the picture the ranking flips, which the simulator below lets you watch happen.

Any model on any machine

The whole document in one instrument. Pick a machine and a model, set the context, and read the verdict: does the working set fit, where does it live, and how fast does it decode. The first two machines are the real boxes from this project; the rest are the reference tiers.

The scenario chips preload the sessions worth replaying: both rounds of the field test (the trap, then the healthy fit), the CPU path with the MoE and with the gpt-oss stretch, the Mac's unified-memory answer, and the used-3090 upgrade path that unlocks the dense 27B. Touch any control and you leave the scenario; the chips get you back.

What each rig comfortably does

The question behind all the numbers: what work can you actually hand this machine? "Comfortable" here means three things at once: the model fits in fast memory, it responds at a usable pace, and the model itself is capable enough for the job. The matrix computes all three from the same formula as the simulator, for every machine and model on this page. Click any cell for the reasoning behind its four verdicts.

Assumptions: q4 weights, q8 KV cache, decode at 65% of the formula ceiling; context per task: 8k short text, 16k coding, 32k long documents and agent loops. Speed bands from the field doctrine (15+ tok/s comfortable, under 10 painful; coding wants 20+). The capability caps (a 4B is not a coding model, no matter the hardware) are the author's judgment, named in the cell details. Everything est; a bench run on your machine (section 13) outranks every cell.

Theory meets an 8 GB card

In August 2026 this document's scenario met a real machine: a small-enterprise Windows box with a Ryzen CPU, 128 GB of RAM, and an RTX 2070 Super with 8 GB of VRAM, driving an agent harness in a thin Linux VM against a network analytics platform over MCP. Everything below was watched happen, not projected.

Round one: the shortlist pick, split across memories. The 30B-A3B MoE (18.6 GB at q4) cannot fit an 8 GB card, so the server split it: a slice of layers in VRAM, the rest in system RAM. It worked, and it crawled. Multi-minute turnsobs, most of it prefill over the harness's ~20k tokens of tool schemas. The split ran at the speed of its slowest memory, and the CPU side of a desktop board is 65 GB/s on a good day.

Round two: a smaller model, all on the card. A dense 4B instruct model at 2.5 GB fits entirely in VRAM with room left for a quantized KV cache. After the cache fix in the next section, the box reached its healthy state: GPU pinned at 100%, CPU at 7%, shared GPU memory at zeroobs. Turns went from painful to workable, and the model still drove the MCP tool chain correctly, which for reporting work is the entire job.

Honesty note: no decode rates were benchmarked on this box yet, so this page still claims no measured tok/s for it. "Painful" and "workable" are the observed states; the bench run in section 13 converts them into numbers.

The shared-memory trap

The single most valuable discovery of the field test. When weights plus KV cache exceed dedicated VRAM on Windows, the NVIDIA driver does not error. It silently pages the overflow into "shared GPU memory," which is ordinary system RAM reached across the PCIe bus. Nothing in the serving logs mentions it. The only symptom is that everything is inexplicably slow, and it is slower than an honest CPU split, because every token now commutes across the bus. The simulator's dashed overflow segment is exactly this state.

The fix on the field-test box was to shrink the KV cache until the whole working set fit on the card:

# Windows host serving via Ollama. Set, then FULLY quit and restart
# the server process (tray quit; verify with: tasklist | findstr ollama)
setx OLLAMA_FLASH_ATTENTION "1"
setx OLLAMA_KV_CACHE_TYPE "q8_0"

If q8 is not enough, the next screws in order: KV cache at q4_0 (halves it again, small quality cost on long contexts), or a smaller serving window. A 22k-token working session fits comfortably under a 32k window; do not pay 64k of cache for headroom you are not using. The KV quant select in the simulator is these screws, live: flip the 4B on the 2070 Super from fp16 to q8 at 64k context and watch the overflow disappear.

The plumbing lies before the model does

Every failure in the field test happened below the model. The model, once fed correctly, just worked. These three cost the most hours.

The GUI beats your environment variables

Ollama's desktop app has its own settings, including a context-length slider that defaults to 4k, and the GUI value overrides OLLAMA_CONTEXT_LENGTH set via setx. The symptom: HTTP 400, "request (20800 tokens) exceeds the available context size (4096 tokens)," while you are certain you configured 65536. The only source of truth is ollama ps and its CONTEXT column, which reports what the loaded instance actually got. Env vars are a request; the GUI is a decision.

Read the suffix before you pull

Model names carry their training recipe: base is a raw text completer, -instruct chats and emits standard JSON tool calls with no thinking phase, -thinking reasons before answering, -coder specializes in code. Dated refreshes (the 2507 style) can split a hybrid line into separate instruct and thinking models and extend context. The expensive gotcha: some coder lines emit an XML tool-call dialect that OpenAI-compatible harnesses cannot parse, while the plain instruct lines of the same family speak standard JSON. If tool calls render as text instead of executing, suspect the dialect before you suspect the harness.

Harness context floors

Some agent harnesses enforce a minimum context window (64k in the one field-tested) and check it against the model's reported maximum, not your serving window. Models reporting 32-40k get refused outright; a 4B that reports 256k sails through. Two numbers to check before pulling gigabytes: what the model reports, and what the harness demands.

Measure before you promise

Every decode number above, slider outputs included, is an estimate until run on the actual machine, because shared cloud VMs deliver whatever bandwidth the neighbors leave. Twenty minutes of measurement turns this whole document from theory into a quote you can stand behind.

# 0. If serving via Ollama: what did the loaded model ACTUALLY get?
ollama ps                                # CONTEXT column = the truth
ollama run <model> --verbose "test"      # eval rate = decode tok/s

# 1. What is this machine? Core count and CPU generation drive prefill.
lscpu | grep -E "Model name|^CPU\(s\)|NUMA"

# 2. Measured memory bandwidth. This one number predicts decode speed.
sysbench memory --memory-block-size=1M --memory-total-size=32G \
  --memory-oper=read --threads=$(nproc) run

# 3. The real test: llama-bench with an agent-sized prompt.
#    pp8192 = prefill tok/s (first-token pain), tg128 = decode tok/s.
llama-bench -m qwen3-coder-30b-a3b-q4_k_m.gguf -p 8192 -n 128 -t $(nproc)

# 4. End to end: serve, then time a realistic harness turn.
llama-server -m model.gguf -c 32768 --jinja &
time curl -s localhost:8080/v1/chat/completions -d @sample-agent-turn.json

Read it like this: tg128 under ~10 tok/s means agent turns will feel broken, look for a better box. 15+ is workable, 25+ is comfortable. If pp8192 is painful, confirm prefix caching is doing its job by sending the same request twice; the second must be dramatically faster. And on any Windows GPU host, keep Task Manager's GPU panel open during the run: dedicated high is healthy, shared above zero is section 11. Once you have a measured bandwidth number, feed it into the calculator in section 01: from then on the page computes with your machine's truth instead of spec sheets.

The call, updated by the field

For a single-user agent querying one data source a few hundred times a day, the field test suggests the smallest model that fits entirely in fast memory and drives tools correctly is very likely the end of the story. The 128 GB was never the constraint, and it turned out the 8 GB was not either, once everything respected it. If a box ever serves a team instead of one agent, the story changes along section 04's roofline: server engines built around continuous batching (vLLM, SGLang, TensorRT-LLM) exist to push decode toward the compute ceiling; for one user, llama.cpp and Ollama remain the right size.