On this page 9
This is the PC half of our Mac Studio M5 Ultra guide. That piece is about a machine with one pool of memory. This one is about what happens on a graphics card when the model is bigger than the card. Every number is ours unless we say otherwise: the same model files and prompts on an RTX 5090, a Radeon AI PRO R9700, an RTX PRO 6000 Blackwell, a DGX Spark, an H100, an H200, a B200 and a MacBook Pro, through our own benchmark harness. Method at the end.
Where the weights live
A language model writes one token at a time, and every token drags the weights it needs out of memory. For a dense model that’s all of them. For a mixture-of-experts model it’s a small, changing subset: DeepSeek V4 Flash stores 284B parameters and uses about 13B per token, picked by a router that decides afresh at every layer, for every token.
The split isn’t arbitrary. llama.cpp’s fit tool keeps on the card everything a token touches every time: attention, the shared layers, the context cache, and the experts of as many layers as will fit. The experts of the remaining layers stay in system RAM, and the CPU does their share of the math. Which experts a token wants changes every token, so that traffic never stops. On a desktop, 2 to 3GB of expert weights cross the bus per token, and the card spends most of each token waiting.
What a spill costs
Here is every split we ran. “In RAM” is roughly how much of the file sat on the CPU side. Writing and reading are at a 4K-token prompt; the last column is how long the first word takes after a 16K-token prompt, which is what an agent hands a model every turn.
| Model, file | Card and the PC behind it | In RAM | Writes | Reads | 16K prompt |
|---|---|---|---|---|---|
| gpt-oss-120b, 63GB | RTX 5090 32GB · 96GB DDR5 | ~33GB | 35 tok/s | 186 tok/s | 77 s |
| gpt-oss-120b | Radeon AI PRO R9700 32GB · 128GB DDR4 | ~35GB | 20 | 174 | 80 s |
| Qwen3.5-122B-A10B, 77GB | RTX 5090 | ~46GB | 22 | 98 | 161 s |
| Qwen3.5-122B-A10B | Radeon AI PRO R9700 | ~49GB | 13 | 101 | 157 s |
| Qwen3.8-Flash-Next, 111GB | Radeon AI PRO R9700 | ~84GB | 11 | 96 | 150 s |
| DeepSeek V4 Flash, 155GB | RTX PRO 6000 96GB · 256GB DDR5 | ~61GB | 22 | 165 | 96 s |
| GLM-5.3-Flash, 157GB | RTX PRO 6000 | ~64GB | 15 | 116 | 128 s |
| MiMo-V2.6-Flash, 167GB | RTX PRO 6000 | ~73GB | 18 | 124 | 138 s |
| MiMo-V2.5, 185GB | RTX PRO 6000 | ~92GB | 13 | 93 | 157 s |
| DeepSeek V4 Flash | H200 NVL 141GB · 1TB DDR5, eight channels | ~17GB | 54 | 347 | 46 s |
| MiMo-V2.6-Flash | H200 NVL | ~29GB | 51 | 252 | 76 s |
| MiMo-V2.5 | H200 NVL | ~47GB | 29 | 157 | 135 s |
| DeepSeek V4 Pro, 850GB | H200 NVL | ~715GB | 4.7 | 7.5 | 1K prompt: 130 s |
For scale, the same gpt-oss-120b file on a card that holds it, an RTX PRO 6000, writes at 212 tok/s and reads at 5,909. The same DeepSeek V4 Flash file whole on a B200 writes at 64 and reads at 812.
The trick saves the writing. It doesn’t save the reading.
Writing survives because a token only needs its few experts, and 2 to 3GB over two DDR5 channels is about 30 milliseconds. Reading collapses because a prompt is thousands of tokens at once, and every one of them wants its own experts from RAM. A card that holds the model reads a 16K prompt in seconds: 3 for gpt-oss-120b on the PRO 6000, 21 for DeepSeek V4 Flash on the B200. Spilled, it takes well over a minute, up to nearly three, and the card sits at about a quarter of its power limit, idling. gpt-oss-120b on the 5090 read at 186 tok/s. The Radeon, a card with a third of the 5090’s bandwidth, read at 174. The PRO 6000, at $18,000, got 165 on DeepSeek. Once the experts live in system RAM, the card is no longer what you’re waiting on.
Two things set the speed instead, and a desktop has little of either. The first is memory bandwidth: two channels of DDR5 move about 90GB/s, and the Radeon’s PC, on DDR4 at about 50GB/s, wrote gpt-oss-120b at 20 tok/s to the 5090 PC’s 35. The second is the cores multiplying those experts. Give the PRO 6000’s PC four cores for that instead of eight and DeepSeek V4 Flash drops from 22 to 15 tok/s on the same two channels. The GPU has 24,000 cores and 1.8TB/s. It’s watching eight cores and 90GB/s do the work.
Dense models get no trick
A dense 70B model needs all of its weights for every token, so there is no small subset to keep on the card. Spill it and every token walks the whole file over the bus. In a comparison of an RTX 5090 PC against the M5 Ultra, a 70B that didn’t fit in 32GB finished over 20× slower on the PC, and even an M5 Max laptop beat it. MacStories got the same collapse from a 27B that normally fits: a long context pushed 16 of its 66 layers into system RAM, and it wrote at 4.6 tok/s with 64K of context loaded, 1.5 at 256K. If your model is dense and bigger than your card, there is nothing to tune. Buy memory or pick a smaller model.
Server memory behind the card
The H200 rows in the table are the same spill with a different PC behind the card: a Xeon with eight DDR5 channels and a terabyte of RAM. DeepSeek V4 Flash doesn’t fit in 141GB either, so 17GB of experts went to RAM. It still wrote at 54 tok/s, which is faster than the Mac Studio’s reviewed 53, and read a 4K prompt at 347 tok/s, twice the desktop spill. MiMo-V2.6-Flash the same way: 51 tok/s with 29GB in RAM, where the PRO 6000 with 73GB in desktop RAM managed 18. The overflow is smaller on the bigger card, and the channels behind it are four times as wide. Both help, and a small overflow into eight channels costs you reading, not writing.
We also pushed it to the silly end. DeepSeek V4 Pro is 1.6 trillion parameters, an 850GB file at 4-bit. The card held 135GB, the server’s RAM held the rest, and it ran: 4.7 tok/s writing, two minutes to read a 1,000-token prompt. Running, technically. At that split 85% of the model lives on the CPU side of the slot, and the GPU mostly watches.
Files that are smaller than they look
Not every file bigger than the card is a spill. Qwen3.8-Flash-Next is a 111GB download that runs whole on a 96GB RTX PRO 6000, and it isn’t a trick. The model is sold as 125B parameters, but it ships another 55B on top, mostly one big n-gram lookup table: 29GB of the file. llama.cpp treats that table like the token embeddings, an input-layer lookup that stays in system RAM on every machine, Mac included, and never goes to the GPU. The 80GB that does the math fits on the card with room for a 64K context.
So the PRO 6000 ran it at 101 tok/s writing and 2,168 tok/s reading at a 4K prompt, and served eight people at once at 198 tok/s between them. The H200 and the B200, with the whole file on the card, wrote it at 87 and 88. (Our Flash-Next runs on the PRO 6000 were on the 600W Workstation Edition; the rest of this article’s PRO 6000 numbers are the 300W Max-Q.) The same file on a 32GB Radeon is a real spill, 84GB in RAM, and the table shows what that costs: 11 tok/s.
Check the download size, then check what’s in it.
Datacenter cards for one person
The question we kept getting: how far is a desk from the hardware the labs use? So we gave an H200 (141GB) and a B200 (180GB) the same files and prompts. For one person, both are an RTX PRO 6000: on gpt-oss-120b the H200 writes at 183 tok/s, the B200 at 199, the PRO 6000 at 212. The B200 has 8TB/s of memory bandwidth; writing Qwen3.8-27B for one person, llama.cpp moves about 1.4 of them. An H100 is slower than a 5090 for one person (55 vs 78 tok/s on Qwen3.8-27B).
Datacenter cards are built for crowds, and that’s where they leave a desk behind: with 64 people on Qwen3.8-27B at once the B200 served 949 tok/s in total, and 1,646 on gpt-oss-120b. The PRO 6000 with eight people on Qwen3.8-27B managed 186. The big cards also idle hot, 92W for the H200 and 173W for the B200, before anyone asks them anything.
If you’re serving a product, rent one of these. If you’re one person with a big model, a $30,000-plus card won’t write your tokens any faster than the $17,000 one.
The office build
Everything above is one box for one person. Put five people on any of them and the Mac Studio flattens, the DGX Spark crawls, and the PRO 6000 runs out of memory for the models worth sharing. The build that fixes this isn’t on anyone’s launch slide, because nobody sells it as a product.
Start with the memory, not the GPU. A current AMD EPYC has twelve DDR5 channels. Twelve 64GB DDR5-6400 sticks give you 768GB at 614GB/s (12 channels × 6,400 MT/s × 8 bytes). That’s half a Mac Studio’s bandwidth with three times the memory, and unlike the Mac, you can put a GPU in front of it. One RTX PRO 6000 gives you 96GB at 1.8TB/s for the parts every token touches (attention, shared layers, the context cache), with 768GB behind it for the experts. GLM-5.3-Flash, MiniMax M3, Qwen3-Coder-480B and the 671B DeepSeeks at 4-bit all fit, with room for context.
Buy the memory channels. The cores come with them.
We know roughly how that behaves, because we ran the cheap version of it: our RTX 5090 with gpt-oss-120b’s experts in desktop RAM, two DDR5 channels, about 90GB/s. It wrote at 35 tok/s, which is almost exactly what the bandwidth math says (5.1B active parameters at 4-bit is about 3GB per token, and 90 ÷ 3 is 30). Twelve channels is seven times that pipe. The same napkin puts DeepSeek V4 Flash (13B active, about 7GB a token) somewhere around 50-80 tok/s on the EPYC box with every expert resident, and the 671B DeepSeeks (37B active, 20GB a token) around 20-30. DeepSeek V4 Flash on the PRO 6000, with 61GB of experts in desktop RAM, got 22 tok/s where this math says 30 to 40, so read the low half of those ranges. Reading, the part that collapsed on our desktops because eight to twelve CPU cores were doing the expert math, gets 128 Zen 5 cores instead. We haven’t built this one yet; those numbers are napkin, anchored to the spill table above.
Three things to know before you spec it. The bandwidth comes from the channels, and only parts with enough chiplets can feed all twelve; the 128-core EPYC 9755 does, the low-core-count bargain parts can’t. The cores are not optional either, as the four-versus-eight test above shows; the parts that feed twelve channels bring plenty. And know the ceiling: this is a 300B-to-700B box, not a DeepSeek V4 Pro box. That one is 850GB at 4-bit, and on a server with 1TB of RAM and an H200 it ran at 4.7 tok/s. But MoE is what lets a small company put a 300B model in a closet instead of on a cloud bill, behind a GPU that doesn’t flatten at four users the way Apple silicon does.
One pool doesn’t spill
A Mac and a DGX Spark have nothing to split: CPU and GPU share one memory. When a model fits, nothing collapses. A single Spark holds gpt-oss-120b whole and reads it at 1,577 tok/s, eight times the spilled 5090, while writing at 57. When a model doesn’t fit, it doesn’t load, and there is no trick. That’s the whole reason the 256GB Mac Studio exists as a product, and the Mac guide linked above is about what it holds.
The little brother shows the shape. Our 16-inch MacBook Pro M5 Pro has 48GB, of which macOS gives the GPU about 37GB, so gpt-oss-120b’s 61GB file doesn’t load at all. What does load runs like this on llama.cpp:
| MacBook Pro M5 Pro, 48GB | Short prompt | 32K prompt | 64K prompt |
|---|---|---|---|
| Qwen3.8-27B (dense) writes | 16 tok/s | 14 tok/s | 12 tok/s |
| Qwen3.8-27B reads | 354 tok/s | 271 tok/s | 200 tok/s |
| Wait before the first word | 3 s | 1.9 min | 5.1 min |
| gpt-oss-20b writes | 86 tok/s | 61 tok/s | 48 tok/s |
| Qwen3.6-35B-A3B writes | 74 tok/s | 57 tok/s | 45 tok/s |
Models up to about 35GB load. Writing is fine for chat. Reading is the problem: a 64K-token prompt is a five-minute wait, which rules out agent work on big codebases. A laptop with a quarter of the Ultra’s hardware landing at a quarter to a third of the Ultra’s reviewed numbers is also how we know those numbers are believable.
What we’d build
| You | Build |
|---|---|
| One person, models under 32GB, want the fastest box | An RTX 5090 in a PC with 96GB of RAM. On the fifteen models that fit both cards it wrote 5% faster than the PRO 6000 Max-Q and read 20% faster, which is what 575W does to the same silicon held to 300W. The RAM is for the spill, which keeps gpt-oss-120b usable at 35 tok/s. Don’t expect it to read fast. |
| One person, models under 90GB, money no object | RTX PRO 6000 in a real PC. The extra $12,000 over a 5090 buys the 96GB, not speed, and 96GB is the point. |
| One person, models over 96GB | A 256GB Mac Studio, or accept 15-22 tok/s on the PRO 6000 with the experts in RAM. The spill is for the occasional big model, not a daily driver. |
| A team of five to twenty | Two DGX Sparks if the models fit in 128GB each. The EPYC box above if they don’t. |
| A product with real traffic | Rent datacenter cards by the hour and stop reading buying guides. |
How we tested
- Engine: llama.cpp release b11146, the prebuilt binaries for each platform (CUDA for NVIDIA, Vulkan for the Radeon, Metal for the Macs). Whole model on the GPU wherever it fits; otherwise the placement llama.cpp’s fit tool chose for a 19K-token context, with the expert tensors of the last layers in system RAM. GLM-5.3-Flash ran on b11429, the first release that knows its architecture.
- Files: the same GGUF on every machine. gpt-oss-120b is the MXFP4 build (63GB on disk), Qwen3.5-122B-A10B Unsloth’s Q4_K_M (77GB), Qwen3.8-Flash-Next Unsloth’s UD-Q4_K_XL (111GB), DeepSeek V4 Flash and V4 Pro Unsloth’s UD-Q4_K_XL (155GB and 850GB), GLM-5.3-Flash Unsloth’s UD-IQ4_XS (157GB), MiMo-V2.5 Unsloth’s MXFP4_MOE (185GB), MiMo-V2.6-Flash the MXFP4 build (167GB), Qwen3.8-27B Unsloth’s UD-Q4_K_M (16.5GB).
- Prompts: a synthetic log-analysis task at 1K, 4K, 16K, 32K and 64K tokens, 256 tokens generated, fresh cache each time. Three runs per size, medians reported. Split runs stop at 16K, because nobody waits longer than that. Crowds are 1, 2, 4 and 8 simultaneous users with a 1K prompt each; the 64-user B200 figures are from llama.cpp’s batched benchmark on the same files.
- The PCs: the RTX 5090 sits behind a Core i9-14900KF with 96GB of DDR5 on two channels; the Radeon AI PRO R9700 behind a Ryzen 9 5900X with 128GB of DDR4, two channels; the RTX PRO 6000 spill runs and Flash-Next ran on the 600W Workstation Edition behind a Core Ultra 9 285K with 256GB of DDR5, two channels; the rest of the PRO 6000 rows are the 300W Max-Q edition. llama.cpp got one thread per P-core on the Intel hosts (eight) and one per core on the Ryzen (twelve). The H200 is the 141GB NVL card in a server with a Xeon Platinum 8592+ and 1TB of DDR5 on eight channels. The MacBook is the 16-inch M5 Pro with 48GB.
- Power: what the GPU driver reports. That’s the card, not the wall.
Sources
- Our own benchmarks (October 2026): llama.cpp on an RTX 5090, a Radeon AI PRO R9700, an RTX PRO 6000 Blackwell in both editions, a DGX Spark, an H100, an H200, a B200 and a MacBook Pro M5 Pro. Method in “How we tested” above.
- Prices (US, October 2026): street-price tracking for the RTX 5090 and the Radeon AI PRO R9700, NVIDIA’s DGX Spark repricing, and our own check of B&H, Newegg and Micro Center listings for the RTX PRO 6000.
- Hardware Canucks: “Apple M5 Ultra vs The Fastest PC” (YouTube). The 70B spill-over test on a 9950X3D + RTX 5090 PC.
- MacStories: M5 Ultra Mac Studio review (Sept 21, 2026). Their RTX 5090 spill numbers.
- Alex Ziskind: “M5 Ultra: Apple wasn’t messing around” (YouTube, Apple loaner). The Mac Studio’s DeepSeek V4 Flash numbers in the chart above.
- Qwen3.8-Flash-Next model card. The 55B of extra parameters and the n-gram table.