15Why Run AI Locally (and What Hardware You Need)
Privacy, zero per-token cost, offline resilience and control — plus an honest guide to RAM, GPUs, model sizes and quantization so you know exactly what your machine can run.
What you'll learn
- Weigh the pros and cons of local vs. cloud AI
- Understand parameters, quantization and memory needs
- Match model sizes to your hardware
- Plan a hybrid local + cloud setup
Why local?
Privacy
Client contracts, medical, legal or financial data never leave your machine. Easier compliance conversations.
No per-token cost
Run thousands of classifications or summaries for the cost of electricity.
Works offline
Brownouts, weak internet, travel — a laptop on battery still has AI.
Control
Pick exact models and versions. No surprise changes, rate limits or account issues.
Local is great for
- Summarizing, classifying, extracting, rewriting
- Private document Q&A (RAG)
- High-volume, repetitive automation steps
- Prototyping and learning without a bill
- Coding help on modest tasks
Cloud still wins for
- The hardest reasoning, analysis and writing
- Long, complex agent runs
- Very long documents (huge context windows)
- Teams without capable hardware
- Zero maintenance
Three numbers that decide what you can run
| Concept | What it means | Rule of thumb |
|---|---|---|
| Parameters | The model's size, e.g. 4B = 4 billion. More parameters ≈ more capable and more memory. | 1–4B: basic tasks · 7–9B: solid all-rounder · 12–32B: strong · 70B+: near-frontier on some tasks |
| Quantization | Compressing weights from 16-bit to 8, 5 or 4 bits. Much smaller, slight quality loss. | Q4 (4-bit) is the sweet spot and Ollama's usual default |
| Memory (RAM/VRAM) | The model must fit in memory, plus room for the context window. | At 4-bit: roughly 0.6–0.7 GB per billion parameters, plus 1–4 GB for context |
What your machine can run
| Your memory | Comfortable model size | Examples (Ollama) | Feels like |
|---|---|---|---|
| 8 GB | 1–4B | gemma3:1b gemma3:4b llama3.2:3b qwen3:4b | Capable helper for short tasks |
| 16 GB | 7–9B | qwen3:8b llama3.1:8b mistral | Good all-rounder, tool calling, RAG |
| 24–32 GB | 12–20B | gemma3:12b qwen3:14b gpt-oss:20b | Strong reasoning, agents, coding |
| 48–64 GB+ | 27–32B+ | gemma3:27b qwen3:32b | Very capable for most business tasks |
Approximate, for 4-bit models with moderate context. New models arrive constantly — check the Ollama library.
CPU, GPU and Apple Silicon
| Hardware | What to expect |
|---|---|
| Apple Silicon Mac (M1–M4) | Excellent. Unified memory lets the GPU use most of your RAM. A 16 GB MacBook runs 8B models smoothly; 32 GB+ runs 14–20B. |
| NVIDIA GPU | Fastest on PCs. Model runs from VRAM (e.g. RTX 3060 12 GB fits ~8–12B at 4-bit). Spills to RAM if too big — slower. |
| AMD GPU | Supported on many recent cards (ROCm on Linux, and on Windows for supported models). Check Ollama's GPU docs. |
| CPU only | Works for small models (1–4B). Expect slower responses; fine for background automation. |
The hybrid setup most pros use
Because Ollama exposes an OpenAI-compatible API, the same code and tools (n8n, Hermes, Open WebUI, your scripts) can switch between local and cloud by changing a URL and model name.
Key takeaways
- Local AI = privacy, zero per-token cost, offline use and control.
- Parameters, quantization and memory decide what runs; 4-bit is the sweet spot.
- 16 GB runs 7–9B models well; 32 GB+ opens up 14–20B models.
- Use a hybrid: local for private/high-volume work, cloud for the hardest tasks.
Knowledge check
0 / 3Q1What does quantization do?
Quantization lowers precision (e.g. to 4-bit) to shrink memory needs.
Q2Comfortable model size for a 16 GB laptop?
At 4-bit, 7–9B models fit well in 16 GB with room for context.
Q3Why is Apple Silicon good for local AI?
Unified memory means large models can run on the GPU.