Local LLM Hardware Guide

Local LLM Hardware Guide

Running ai/claw plugins fully offline.

When to consider local models

By default, the ɳSelf ai plugin routes through the Multi-AI Router (MAR), which prefers in this order:

1. Local models: if configured and reachable.
2. Gemini Free Pool (GFP): Google AI free tier.
3. Subscriptions: Claude Max, ChatGPT Plus reuse.
4. Pay-per-token APIs: OpenAI, Anthropic, Google.

Local is one option among four, not the default. Choose it when you need data residency, predictable cost on heavy workloads, or air-gapped operation. For most users, GFP plus an API fallback is cheaper and faster.

Hardware tiers

Numbers below assume Q4 quantization unless noted. Throughput is single-user batch=1.

Minimum. Llama 3.1 8B (Q4)

• VRAM: 6-8 GB (RTX 3060 12 GB, RTX 4060, M1/M2 base 8 GB unified)
• Throughput: 30-60 tok/s on consumer GPU; 8-15 tok/s on Apple Silicon base; ~2 tok/s CPU-only fallback
• Use case: routing, classification, short summaries, ai-cost-router task classes

Recommended. Llama 3.1 70B (Q4) or Qwen 2.5 32B (FP16)

• VRAM: 24-48 GB (RTX 4090 24 GB tight, dual 3090, M3 Max 64 GB unified, M2 Ultra 64 GB)
• Throughput: 15-25 tok/s
• Use case: general assistant work, claw email triage, code completion. Roughly matches GPT-4o-mini quality at most tasks.

Power-user. Llama 3.1 405B (Q4) / Mixtral 8x22B (FP16) / Qwen 2.5 72B (FP16)

• Hardware: Mac Studio M2 Ultra 192 GB OR multi-GPU rig (4× RTX 4090 / 2× A6000 / H100 80 GB)
• Throughput: 5-15 tok/s
• Use case: tasks where GPT-4-class reasoning matters offline. Most users should reach for the API instead.

Software stack

• Ollama: recommended for single-user. One-line install, OpenAI-compatible endpoint at localhost:11434. Configure MAR with OLLAMA_BASE_URL.
• llama.cpp: lower-level, broader hardware support (CPU, Metal, CUDA, Vulkan, ROCm). Use when Ollama doesn’t fit.
• vLLM: multi-user inference with batching. Use for shared team deployments; requires CUDA.

MAR configuration lives in services/ai/config.yaml. Point a local provider at the Ollama or vLLM endpoint and set its priority.

Cost of ownership

Rough comparison for a workload of 50M tokens/month (mixed input/output):

Path	Monthly cost	Notes
OpenAI gpt-4o-mini API	~$15-30	No capex, no ops
Anthropic Claude Haiku API	~$20-40	No capex, no ops
Local 70B on RTX 4090 (used $1,200)	~$8 power + amortized $33/mo over 36 mo	Crosses break-even vs API around 12-18 mo at this volume
Local 8B on existing M3 Pro Mac	~$2 power	Already-owned hardware, near-zero marginal cost

API wins for low and bursty volume. Local wins for steady high volume, regulated data, or air-gapped use. The router lets you mix.

Benchmarks and references

• llama.cpp throughput tables: https://github.com/ggerganov/llama.cpp/discussions (search “performance”)
• Anyscale LLM benchmarks: https://www.anyscale.com/blog/llama-2-is-about-as-factually-accurate-as-gpt-4-for-summaries-and-is-30x-cheaper
• Ollama model registry with sizes and quantizations: https://ollama.com/library

Numbers shift quickly as new model releases land. Treat the table above as a starting point and verify on your hardware before committing to a tier.

See also

• Local LLM hardware recommendations: per-model VRAM/throughput matrix, cooling/PSU/chassis guidance, cloud GPU rental costs
• AI cost optimization: how MAR routes between providers
• Configuration: ai plugin settings