Running a model on your own hardware: what it really takes

The real question isn’t “can it run”

Every few weeks someone asks me if they should just buy a GPU and run models locally instead of paying for API access. The honest answer is that almost anything can run a model if you’re willing to wait. A 7B model will limp along on a laptop CPU. The real question is whether it runs fast enough and with enough context to be useful for what you’re building, and that’s where most people underestimate what they’re signing up for.

I run local models for specific jobs: batch document processing where I don’t want data leaving my network, and testing prompt changes against an open weight model before I burn API credits on the hosted version. I still pay for hosted APIs for anything that needs top-tier reasoning or long context. Local hardware didn’t replace that spend, it just changed which jobs I route where.

Where the memory goes

The number that decides everything is VRAM, and it goes to two places: the model weights and the KV cache.

Weights are the easy part to estimate. A model’s size in memory is roughly its parameter count times the bytes per parameter. At 16-bit precision, a 7B parameter model needs about 14GB just to hold the weights. At 8-bit, that drops to roughly 7GB. At 4-bit, roughly 4GB. This is why quantized models dominate the local scene: a 4-bit quant turns a model that needs a 24GB or 48GB card into one that fits on a 12GB card.

The part people forget is the KV cache, the memory that holds attention keys and values for every token in your context window. That grows with context length, batch size, and the number of attention heads and layers in the model. Feed a local model a long document or a big system prompt and you can watch VRAM climb even though the weights themselves haven’t changed. This is the reason a model that “fits” at a 2k context window can run out of memory at 32k. If you’re building anything with real context, you have to budget for the cache, not just the weights.

Bandwidth, not just capacity, sets your speed

Capacity tells you whether a model fits. Memory bandwidth tells you how fast it runs.

Autoregressive decoding, generating one token at a time, is memory bound, not compute bound. For every token, the GPU has to read the entire set of weights (or the active weights, for mixture-of-experts models) from memory. That means generation speed is limited by how many gigabytes per second your memory can move, divided by how many bytes you need to move per token. This is why a card with more raw compute but the same memory bandwidth doesn’t generate text meaningfully faster. It’s also why high-bandwidth memory, not core count, is the spec that actually matters for chat-style inference.

This is the single biggest thing people miss when they price out hardware. They look at VRAM capacity and stop there, then wonder why their setup fits a large model but produces text slower than they can read a paragraph.

Quantization: the lever everyone pulls

Quantization is the main tool for making local hardware viable, and it’s a real tradeoff, not a free lunch. Formats like GGUF, GPTQ, and AWQ compress weights from 16-bit down to 8-bit, 4-bit, or lower, cutting memory footprint roughly in proportion. The cost is quality degradation, and it isn’t uniform. Some models tolerate 4-bit quantization with barely noticeable loss on general tasks. Others show clear degradation on tasks that need precision, like arithmetic, code generation, or following multi-step instructions closely. I don’t have controlled benchmark numbers to hand you for exactly how much a given model degrades at a given quant level, and I’d be skeptical of anyone who states one without having run the comparison themselves on their own task. What I can tell you from building with these tools is that you should test your actual task against the quantized model before committing to it, not assume the marketing claim of “near-lossless” holds for your use case.

What consumer hardware actually gets you

On the consumer side, the practical ceiling for a single card is VRAM capacity. Cards with 24GB of VRAM, which includes both current and previous-generation high-end consumer GPUs, comfortably hold 4-bit quantized models in the 30B to 70B parameter range with some room for context. Cards with 12GB to 16GB handle smaller models well, generally in the 7B to 13B range at 4-bit, before context eats the headroom. Below that, you’re mostly running small models or accepting heavy quantization and short context windows.

None of this is exotic hardware. It’s the same GPUs people use for gaming or video editing, which is exactly why local LLM hosting became a hobby in the first place: the hardware was already sitting on a lot of desks. The catch is that gaming cards weren’t designed with LLM memory bandwidth as the priority, so you inherit whatever bandwidth that card’s memory bus happens to have.

The Mac question

Apple Silicon comes up constantly because unified memory lets a Mac address a much larger pool for the model than a discrete GPU’s VRAM would allow at a comparable price. A machine with a large unified memory pool can hold a bigger model than a 24GB GPU can. The tradeoff is memory bandwidth. Apple’s unified memory is fast for a laptop chip, but it’s still generally well below the bandwidth of a dedicated high-end GPU’s memory bus. So the shape of the tradeoff flips: Macs tend to fit bigger models but generate more slowly, GPUs tend to fit smaller models but generate faster. Which one is right for you depends entirely on whether your bottleneck is “the model doesn’t fit” or “the model isn’t fast enough,” and those are different problems with different fixes.

Multi-GPU is where the complexity spikes

Once a model doesn’t fit on one card, you’re in multi-GPU territory, and that’s where local hosting stops being a weekend project. You need enough PCIe lanes to feed each card, software that supports tensor or pipeline parallelism across devices, and a power supply and case that can handle the draw and heat of multiple 300+ watt cards running continuously. Consumer motherboards often can’t give every slot full-bandwidth lanes, which throttles the inter-GPU communication that parallel inference depends on. This is the point where people who started with “I’ll just buy a GPU” end up pricing out server-grade boards, extra PSUs, and better airflow, and the total cost stops looking like a hobby purchase.

The software stack matters more than people think

Hardware only gets you halfway. The runtime you use, llama.cpp, Ollama, vLLM, or something else, determines how well you actually use the memory and bandwidth you have. Some runtimes support better batching, better quantization formats, or better memory management for long context than others, and the gap between a well-tuned and poorly-tuned setup on identical hardware can be significant. If you’re evaluating local hardware, budget time to also evaluate the runtime, because a bad runtime choice will make good hardware look worse than it is.

The cost math nobody likes to do

The honest pitch for local hardware isn’t that it’s cheaper per token than an API. For low, bursty usage, it usually isn’t, because you’re paying for idle GPU time, electricity, and your own time spent maintaining the stack, none of which show up on an API invoice. The case for local hardware is control: no rate limits, no data leaving your network, no dependency on a vendor’s uptime or pricing changes, and a fixed cost ceiling if your volume is high and steady enough to make that ceiling worth hitting. If your usage is spiky or experimental, hosted APIs are still the pragmatic default. If you have a steady, high-volume, privacy-sensitive workload, that’s when the hardware math starts to make sense.

If you’re trying to figure out where your own workload lands on that line, that’s the kind of decision we break down regularly on AI Tool Gazette, without anyone paying for a spot on the list. Check out more of our coverage on the home page.