Jamesob's guide to running SOTA LLMs locally
The guide provides step-by-step instructions for running state-of-the-art large language models on local hardware, covering setup, model selection, and optimization techniques for users who want to avoid cloud-based AI services.
Background
- This is a practical guide by software engineer James O'Beirne for running "state-of-the-art" large language models (LLMs) like Llama or Mistral on your own computer, rather than relying on cloud services like ChatGPT or Claude.
- The key technical barrier to local LLMs is GPU VRAM: modern models are too large for typical consumer hardware. The guide explains quantization (compressing models to reduce memory needs) and suggests hardware setups (e.g., an M-series Mac with unified memory or a PC with a high-VRAM GPU like the NVIDIA RTX 4090 24GB).
- Running locally matters for privacy (no data sent to third parties), offline access, no usage fees, and avoiding censorship or API restrictions. Downside: local models still lag behind the best cloud models in capability, and good hardware costs thousands of dollars.