Sizing a local model deployment
Memory capacity is the binding constraint, not compute. What fits in what, and what it costs to own.
- Revised
- 1 September 2026
- Recheck by
- 1 December 2026
- What would make this wrong
- Open weight model releases and accelerator pricing. Both move fast. Every price below is a list price read in the month of revision, not a quote.
The first question anyone asks about running a model on their own hardware is how fast it will be. That is the second question. The first is whether it fits in memory at all, because a model that does not fit does not run slowly, it runs at disk speed, which is not a usable answer.
So sizing starts from memory capacity and works outward. A model quantised to four bits needs roughly half a gigabyte of memory per billion parameters, plus headroom for the KV cache, which grows with context length and is the part people forget. Long context is not free. A 262k token window costs real memory that has to sit next to the weights.
What the current open weight tier needs
Qwen3-Coder-Next, released 3 February 2026 under Apache 2.0, is the strongest openly licensed agentic coding model we are aware of at time of writing. It is a mixture of experts design, 80 billion parameters total with roughly 3 billion active per token, which is why it runs faster than its size suggests. At four bit quantisation it occupies about 46 GB, with a 262k native context window.
That single number sets the bracket. 46 GB of weights plus cache does not fit on a 32 GB card. It fits comfortably in 128 GB of unified memory. This is the whole sizing decision in one line, and it is why the table below is organised by memory rather than by price or by teraflops.
| Platform | Memory | Indicative price | What it holds |
|---|---|---|---|
| NVIDIA RTX 5090 | 32 GB | USD 3,000 to 5,000 | Mid sized models at 4 bit. Will not hold a dense 70B. |
| NVIDIA DGX Spark | 128 GB | USD 4,699 | The 46 GB class with room for long context. |
| Apple MacBook Pro M5 Max | 128 GB | USD 3,899 | Same class, portable. Roughly 12 to 18 tokens per second at 70B. |
| Apple Mac Studio M3 Ultra | 512 GB | USD 5,299 and up | Headroom for larger models or several at once. |
| NVIDIA RTX PRO 6000 | 96 GB | USD 12,000 to 14,500 | When throughput per watt matters more than capital cost. |
The number that decides the business case
A 128 GB machine at around USD 4,000 is the entry point for serious local agentic work. Set against cloud inference, the payback arithmetic is simple and worth doing honestly before anyone commits: at current frontier API pricing of roughly USD 5 in and USD 25 out per million tokens, a single machine pays for itself somewhere between 150 and 300 million output tokens, depending on the model you would otherwise have called.
For an individual developer that is a long time. For an agentic workload running loops overnight, or for a team of ten, it is weeks. The case for local inference is almost never made on a single user, and any vendor telling you otherwise has not done the arithmetic.
What we do not know
We have not independently benchmarked the throughput figures in the table above on UAE ambient conditions. Sustained inference is a thermal load, and a machine specified for 25 degrees behaves differently in a room running at 40. That is measurable and we intend to measure it. Until we have, treat the tokens per second column as the vendor is reporting it, which is to say optimistically and at room temperature.