How to run an LLM locally in 2026: a complete guide

Every API call to OpenAI or Anthropic sends your prompt to someone else’s server and charges you per token. Most of the time that’s fine. But if you’re iterating on a coding assistant all day, drafting anything that touches client data, or building a side project you don’t want to run a metered bill on, a local model removes both the cost and the data-leaves-my-machine problem in one move.

This is for operators and developers who are comfortable with a terminal and already have a laptop or desktop with a decent GPU or enough RAM. You don’t need to be an ML engineer. You do need to accept upfront that a 7-14B local model will not match GPT-5 or Claude on hard reasoning tasks. What it’s genuinely good for: autocomplete-grade coding help, drafting and rewriting, summarizing documents that shouldn’t leave your machine, and testing agent workflows without burning API credits.

By the end of this you’ll have a model downloaded and running, a way to chat with it in a browser, and an OpenAI-compatible API endpoint your own scripts can hit, exactly like api.openai.com but pointed at localhost.

what you need

• a machine with 16GB+ RAM (32GB recommended) for smooth 7-8B model inference, or a GPU with 12GB+ VRAM for real speed (RTX 3060 12GB, RTX 4070, RTX 4090, or an Apple Silicon Mac with 16GB+ unified memory)
• 20-50GB free disk space per model you download, GGUF-quantized weights for a 7B model run 4-8GB, a 70B model can run 40GB+
• Ollama, free, the fastest path to a working setup on Windows, macOS, or Linux
• optionally, llama.cpp if you want manual control over quantization and build flags
• Docker Desktop if you want a proper chat UI instead of the terminal
• a model to pull: Llama 3.1 8B, Qwen2.5 7B or 14B, or Mistral 7B are all free, open-weight downloads
• 30-60 minutes for first setup, most of it spent waiting on the model download
• $0 if you’re using hardware you already own. If you’re buying a GPU specifically for this, budget $1,500-2,500 for something like an RTX 4090

step by step

1. check what your hardware can actually run

Before downloading anything, know your ceiling. On Windows or Linux with an NVIDIA GPU, check VRAM:

nvidia-smi

Look at the “Total” memory figure. As a rough rule: a 7-8B model quantized to 4-bit needs about 5-6GB VRAM, a 14B model needs about 10GB, a 70B model needs 40GB+ or a multi-GPU setup. On a Mac, unified memory is shared between CPU and GPU, so a Mac with 32GB can comfortably run a 14B model.

If it breaks: no nvidia-smi output means either you don’t have an NVIDIA GPU or drivers aren’t installed. You can still run smaller models on CPU with Ollama, just expect single-digit tokens per second instead of 30-50+.

2. install Ollama

Ollama wraps llama.cpp with a simple CLI and handles quantization and model management for you. Grab it from the official download page. On macOS or Linux you can also run:

curl -fsSL https://ollama.com/install.sh | sh

On Windows, run the installer, it sets up a background service automatically.

If it breaks: if the service doesn’t start, check that port 11434 isn’t already in use by another process (netstat -ano | findstr 11434 on Windows).

3. pull and run your first model

ollama pull llama3.1:8b
ollama run llama3.1:8b

The pull downloads the quantized weights (about 4.7GB for this one). The run command drops you into an interactive chat in your terminal. Expected output: a >>> prompt where you can type a question and get a streamed response within a few seconds.

If it breaks: a slow or stalled download usually means a flaky connection, ctrl-C and rerun ollama pull, it resumes rather than starting over.

4. benchmark your actual throughput

Numbers on paper don’t tell you what you’ll get. Ask the model something and watch the token stream speed, or check Ollama’s built-in stats:

ollama run llama3.1:8b --verbose

This prints tokens/second after each response. On an RTX 4090 you should see 60-90 tok/s for an 8B model at 4-bit quant. On CPU-only, expect 5-15 tok/s depending on your core count.

If it breaks: if throughput is far below these ranges, confirm Ollama is actually using your GPU, run ollama ps while a chat is active and check the “PROCESSOR” column says GPU rather than CPU.

5. add a proper chat interface

The terminal gets old fast. Open WebUI gives you a ChatGPT-style browser interface pointed at your local Ollama instance:

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data --name open-webui \
  ghcr.io/open-webui/open-webui:main

Open http://localhost:3000, create a local account, and it auto-detects any models you’ve pulled with Ollama.

If it breaks: if Open WebUI can’t see your models, confirm Ollama is running on the host (ollama list) and that the container’s host-gateway mapping is intact, this is the most common failure on Windows with Docker Desktop’s networking modes.

6. expose an OpenAI-compatible API

This is the part that matters if you’re building anything. Ollama serves an OpenAI-compatible endpoint automatically on port 11434:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:8b",
    "messages": [{"role": "user", "content": "say hi in one sentence"}]
  }'

Expected output: a JSON response in the same shape as the OpenAI Chat Completions API. Any SDK or script that talks to OpenAI can point base_url at http://localhost:11434/v1 and work with minimal changes.

If it breaks: a connection refused error means the Ollama service isn’t running, check with ollama list first, it auto-starts the server if needed.

7. pick the right quantization for your VRAM

Ollama defaults to a reasonable quant (usually Q4_K_M), but if you’re VRAM-constrained or want higher fidelity, you can pull specific quant levels. Browse available GGUF variants for a given model on Hugging Face before deciding. Lower bit-depth (Q4) means smaller memory footprint and faster inference at a small quality cost, higher bit-depth (Q8, fp16) means better output at roughly double the memory.

If it breaks: if a model won’t load and you see an out-of-memory error, drop to a smaller quant or a smaller parameter count before assuming your hardware can’t run local LLMs at all.

8. lock it down before exposing it beyond localhost

By default Ollama binds to localhost only, which is safe. If you set OLLAMA_HOST=0.0.0.0 to reach it from another device on your network, you’ve just opened an unauthenticated API to anyone on that network. Put it behind a reverse proxy with basic auth (Caddy or nginx) or a firewall rule restricting the port to trusted IPs before doing this on anything other than a fully isolated home network.

If it breaks: if you can’t reach it from another machine, check your OS firewall isn’t blocking port 11434 inbound, that’s the usual culprit, not Ollama config.

9. keep it running as a background service

On Windows and macOS, Ollama installs as a background service by default and starts on login. On Linux, verify it’s enabled:

sudo systemctl enable --now ollama

If it breaks: if it’s not surviving reboots, check systemctl status ollama for the failure reason, missing GPU drivers after a kernel update is a common cause on Linux.

common pitfalls

Buying a GPU before checking what you actually need. A lot of people jump straight to an RTX 4090 when an RTX 4070 12GB or even a Mac they already own would run their target model fine. Match the GPU to the model size, not the other way around.

Ignoring context window memory cost. VRAM usage isn’t just the model weights, the context window (chat history, RAG chunks) consumes additional memory as it grows. A model that loads fine with a short prompt can run out of memory once you feed it a long document.

Treating a local 8B model as a GPT-5 replacement for everything. Local models are genuinely strong at narrow, well-scoped tasks, weak at long multi-step reasoning and up-to-date knowledge. Use them for what they’re good at and keep a hosted API for the tasks that need frontier-level reasoning.

Running unquantized full-precision weights on consumer hardware. This is the fastest way to hit an out-of-memory error. Quantized (GGUF) formats are the default for a reason, most operators can’t tell the difference between Q4 and fp16 output on everyday tasks.

Skipping the security step when exposing the API. An open, unauthenticated Ollama endpoint on your home network is a real exposure if any other device on that network is compromised. This isn’t hypothetical, it’s the same class of issue covered on theprivacywire.com around self-hosted service exposure generally.

scaling this

Going from personal use to a small team (roughly 10x traffic): a single Ollama instance handles one request at a time reasonably well but queues under concurrent load. Put a lightweight reverse proxy in front and consider OLLAMA_NUM_PARALLEL to allow a few concurrent requests, still on one machine.

Going from a small team to serving a product feature (roughly 100x): a single consumer GPU won’t keep up with real concurrent load. This is where you move to a proper inference server like vLLM, which supports continuous batching and can serve many simultaneous requests off the same GPU far more efficiently than Ollama’s request-at-a-time model. You’re now also thinking about multi-GPU setups or a dedicated inference box.

Going from a product feature to production scale (roughly 1000x): local hardware stops making economic sense at this point, both because of the hardware cost and because you now need redundancy and uptime guarantees a single machine in your office can’t provide. Most operators at this scale rent GPU capacity (RunPod, Lambda, or similar) or fall back to a hosted API for the production path, keeping local inference for development, testing, and anything touching data that genuinely can’t leave the building.

where to go next

Once you’ve got a model running locally, the next question is usually how to make it useful beyond chat. Read how to build an AI agent that uses tools to wire your local endpoint into an actual workflow. If you’re weighing whether to run local for cost reasons specifically, how to cut your LLM API bill in half covers the tradeoffs against hosted APIs directly. And if you plan to feed your local model your own documents, how to choose an embedding model in 2026 is the natural next step before you build a RAG pipeline. For the full archive, see the blog index.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-07-19.