Running a Local LLM for Coding: When It's Worth the Setup
A model running on your own machine trades raw capability for privacy, offline access, and zero per-token cost — worth it for some workflows, not others.
Every AI coding assistant discussed elsewhere on this site runs against a hosted API — your code and prompts leave your machine and reach a vendor's servers. A local LLM is the alternative: a model small enough to run on your own hardware, using a tool like Ollama or LM Studio, with nothing sent anywhere.
The case for it is specific, not universal. If you work with code you're contractually barred from sending to a third party, or on a codebase where an offline environment isn't optional, a local model is often the only option that clears that bar at all — a hosted API simply isn't on the table, regardless of how good it is. It's also genuinely free to run once you have the hardware, with no per-token billing and no rate limit beyond your own machine's throughput.
The honest tradeoff is capability. A model that fits comfortably on a consumer GPU is smaller than the flagship hosted models, and it shows on harder, multi-step coding tasks — a local model is a reasonable substitute for straightforward completions and single-file edits, and a noticeably weaker one for the kind of long, multi-file agentic reasoning the hosted flagship models are specifically tuned for.
Hardware is the other real constraint: a model that runs well needs enough VRAM to hold it, and a card with 8GB will run a meaningfully smaller model than one with 24GB. Anyone deciding whether to go this route should check a specific model's stated memory requirements against their own GPU before assuming it'll run at a usable speed at all.
The practical rule: reach for a local model when privacy or offline access is a hard requirement, not a preference, and accept the capability gap as the cost of clearing that bar. For everything else, a hosted assistant remains the more capable and lower-effort option.