Local LLM Hardware Guide
Local LLM Hardware Guide
Section titled “Local LLM Hardware Guide”Running ai/claw plugins fully offline.
When to consider local models
Section titled “When to consider local models”By default, the ɳSelf ai plugin routes through the Multi-AI Router (MAR), which prefers in this order:
- Local models: if configured and reachable.
- Gemini Free Pool (GFP): Google AI free tier.
- Subscriptions: Claude Max, ChatGPT Plus reuse.
- Pay-per-token APIs: OpenAI, Anthropic, Google.
Local is one option among four, not the default. Choose it when you need data residency, predictable cost on heavy workloads, or air-gapped operation. For most users, GFP plus an API fallback is cheaper and faster.
Hardware tiers
Section titled “Hardware tiers”Numbers below assume Q4 quantization unless noted. Throughput is single-user batch=1.
Minimum. Llama 3.1 8B (Q4)
Section titled “Minimum. Llama 3.1 8B (Q4)”- VRAM: 6-8 GB (RTX 3060 12 GB, RTX 4060, M1/M2 base 8 GB unified)
- Throughput: 30-60 tok/s on consumer GPU; 8-15 tok/s on Apple Silicon base; ~2 tok/s CPU-only fallback
- Use case: routing, classification, short summaries, ai-cost-router task classes
Recommended. Llama 3.1 70B (Q4) or Qwen 2.5 32B (FP16)
Section titled “Recommended. Llama 3.1 70B (Q4) or Qwen 2.5 32B (FP16)”- VRAM: 24-48 GB (RTX 4090 24 GB tight, dual 3090, M3 Max 64 GB unified, M2 Ultra 64 GB)
- Throughput: 15-25 tok/s
- Use case: general assistant work, claw email triage, code completion. Roughly matches GPT-4o-mini quality at most tasks.
Power-user. Llama 3.1 405B (Q4) / Mixtral 8x22B (FP16) / Qwen 2.5 72B (FP16)
Section titled “Power-user. Llama 3.1 405B (Q4) / Mixtral 8x22B (FP16) / Qwen 2.5 72B (FP16)”- Hardware: Mac Studio M2 Ultra 192 GB OR multi-GPU rig (4× RTX 4090 / 2× A6000 / H100 80 GB)
- Throughput: 5-15 tok/s
- Use case: tasks where GPT-4-class reasoning matters offline. Most users should reach for the API instead.
Software stack
Section titled “Software stack”- Ollama: recommended for single-user. One-line install, OpenAI-compatible endpoint at
localhost:11434. Configure MAR withOLLAMA_BASE_URL. - llama.cpp: lower-level, broader hardware support (CPU, Metal, CUDA, Vulkan, ROCm). Use when Ollama doesn’t fit.
- vLLM: multi-user inference with batching. Use for shared team deployments; requires CUDA.
MAR configuration lives in services/ai/config.yaml. Point a local provider at the Ollama or vLLM endpoint and set its priority.
Cost of ownership
Section titled “Cost of ownership”Rough comparison for a workload of 50M tokens/month (mixed input/output):
| Path | Monthly cost | Notes |
|---|---|---|
| OpenAI gpt-4o-mini API | ~$15-30 | No capex, no ops |
| Anthropic Claude Haiku API | ~$20-40 | No capex, no ops |
| Local 70B on RTX 4090 (used $1,200) | ~$8 power + amortized $33/mo over 36 mo | Crosses break-even vs API around 12-18 mo at this volume |
| Local 8B on existing M3 Pro Mac | ~$2 power | Already-owned hardware, near-zero marginal cost |
API wins for low and bursty volume. Local wins for steady high volume, regulated data, or air-gapped use. The router lets you mix.
Benchmarks and references
Section titled “Benchmarks and references”- llama.cpp throughput tables: https://github.com/ggerganov/llama.cpp/discussions (search “performance”)
- Anyscale LLM benchmarks: https://www.anyscale.com/blog/llama-2-is-about-as-factually-accurate-as-gpt-4-for-summaries-and-is-30x-cheaper
- Ollama model registry with sizes and quantizations: https://ollama.com/library
Numbers shift quickly as new model releases land. Treat the table above as a starting point and verify on your hardware before committing to a tier.
See also
Section titled “See also”- Local LLM hardware recommendations: per-model VRAM/throughput matrix, cooling/PSU/chassis guidance, cloud GPU rental costs
- AI cost optimization: how MAR routes between providers
- Configuration:
aiplugin settings