Local LLM Guide
Local LLM Guide
Section titled “Local LLM Guide”ɳClaw runs language models directly on your device. No API keys. No per-message cost. No conversation data leaving your machine.
This guide covers the full local-model workflow shipped in S19: searching HuggingFace, downloading a GGUF file, understanding quantisation options, reading inference metrics, monitoring memory, and swapping models.
Finding a model
Section titled “Finding a model”Open Settings → Local AI. The search box queries HuggingFace for publicly available GGUF-format models.
Type a model name — for example Llama 3.2 3B, Qwen 2.5 0.5B, or mistral-7b — and press Search or Enter. Results appear in a scrollable list showing:
- Model name and author
- Total download count (higher usually means better-tested)
- Number of available GGUF quantisation variants
The search uses the HuggingFace Models API with a 5-minute cache. Queries are unauthenticated and go to HuggingFace’s public endpoint.
Choosing a quantisation
Section titled “Choosing a quantisation”Click a result to select it. A file selector appears below the result list, defaulting to Q4_K_M. This is the recommended starting point for most hardware: a 4-bit quantisation that balances size, inference speed, and output quality.
| Quantisation | Relative size | Quality | Best for |
|---|---|---|---|
Q2_K | ~0.5× | Reduced | Minimal storage or RAM |
Q4_K_M | 1× (default) | High | Most hardware — good balance |
Q5_K_M | ~1.2× | Very high | More RAM available |
Q8_0 | ~2× | Near-lossless | Plenty of RAM, maximum quality |
F16 | ~4× | Lossless | Research or benchmarking |
The file size is shown next to each option. Make sure you have enough free disk space and RAM before downloading.
Downloading
Section titled “Downloading”Click Download & Load. ɳClaw:
- Starts downloading the GGUF file to its local models directory.
- Shows a progress bar with download speed and estimated time.
- Verifies the SHA-256 checksum after download completes.
- Loads the model into memory.
If the download is interrupted (network drop, app quit), it resumes from where it left off on the next attempt via HTTP Range requests. The partial file is stored as <filename>.gguf.part and renamed on successful completion.
You can cancel an in-progress download at any time from the download queue panel.
Inference metrics: TPS and TTFT
Section titled “Inference metrics: TPS and TTFT”Once a model is loaded, two metrics appear below each assistant response in the chat view:
- TPS — tokens per second. How fast the model generates text. Typical values: 15–30 tok/s on modest hardware, 60–100 tok/s on Apple Silicon with Metal acceleration.
- TTFT — time to first token in milliseconds. How long the model takes to begin responding after you send a message. Values under 500 ms feel instant; values above 2000 ms feel slow.
These metrics update after every response. Use them to judge whether a model fits your workflow — or whether to try a smaller quantisation.
Memory monitor
Section titled “Memory monitor”The status bar shows how much RAM (and GPU VRAM on supported hardware) the active model consumes.
| Platform | What is reported |
|---|---|
| macOS (Apple Silicon) | Unified memory via mach_task_basic_info |
| Linux with NVIDIA GPU | GPU VRAM via NVML + system RAM |
| Linux CPU-only / other | System RAM via sysinfo |
If available memory is below the model’s estimated footprint, ɳClaw warns you before loading. The warning includes the estimated model size and current free memory.
Swapping models
Section titled “Swapping models”Loading a new model unloads the previous one automatically. The swap is synchronous: the old model’s memory is fully released before the new model initialises. You will not accumulate memory across swaps.
The swap flow:
- Download the new model (or select one already in your models directory).
- Click Download & Load (or Load if already downloaded).
- The button shows Loading model… while the old model is dropped and the new one initialises.
- Once ready, the chat input becomes active and the new model name appears in the status bar.
Supported hardware
Section titled “Supported hardware”| Platform | Acceleration |
|---|---|
| macOS (Apple Silicon) | Metal — GPU layers configurable in Settings → Local AI |
| macOS (Intel) | CPU only |
| Linux | CUDA (NVIDIA), Vulkan (AMD/Intel), or CPU |
| Windows | CUDA, Vulkan, or CPU |
| iOS / Android | CPU (coming in a future release) |
ɳClaw uses llama.cpp as its inference engine via the llama-cpp-2 Rust crate. The number of GPU layers offloaded is set automatically based on available VRAM and can be overridden in Settings → Local AI → Advanced.
Context window and seed
Section titled “Context window and seed”Two advanced options are available under Settings → Local AI → Advanced:
- Context size — token count for the model’s context window. Default: 2048. Larger contexts use more RAM.
- Seed — random seed for reproducible outputs. Default: 0 (random). Set a fixed value to get the same output for the same prompt (useful for testing).
Tier auto-detection
Section titled “Tier auto-detection”If you are running ɳClaw for the first time, a 60-second background benchmark selects the appropriate model tier for your hardware. See How Local AI Works for the full tier matrix.
The local-LLM features documented here let you search for any GGUF model on HuggingFace and download it directly — bypassing the auto-tier system if you prefer a specific model.
Privacy
Section titled “Privacy”- Models run entirely on-device.
- No conversation text leaves your machine while using a local model.
- Model files are stored in the ɳClaw models directory. On macOS this is
~/Library/Application Support/io.nself.nclaw/models/. - Cloud Model Fallback (Settings → Privacy) is off by default. Enabling it sends conversations to your configured nSelf backend when a local model is unavailable.
Troubleshooting
Section titled “Troubleshooting”Download stalls at 0%: Check your network connection. The downloader contacts HuggingFace’s CDN directly. Some corporate proxies block large binary downloads.
“Not enough memory” warning: Select a smaller quantisation (Q4_K_M → Q2_K) or close other applications to free RAM.
Model loads but generates nonsense: The GGUF file may be corrupted. Delete it from the models directory and re-download.
Very slow inference on macOS Apple Silicon: Confirm Metal acceleration is enabled in Settings → Local AI. If GPU layers is set to 0, inference runs on CPU only.
“Failed to load model” error: Verify the file is a valid GGUF by checking the first 4 bytes: they should spell GGUF. ɳClaw validates this before loading and reports an error if the magic bytes are wrong.