Every AI article on this site, including the one where I compared the three big $20 subscriptions, has the same little secret hiding in it: the AI lives in somebody else’s building. You rent it by the month, it can change its prices, its limits, or its personality whenever the company feels like it, and everything you type takes a road trip to a data center. So I’m doing the most stubborn thing a PC owner can do in 2026. I’m putting an AI inside my own machine, on my own drive, that answers to nobody. No subscription. No cloud. No five hour usage window telling me to come back later like a kid at the arcade who ran out of coins.
This is the plan, the math, and the honest trade, before I build it. If I’m going to spend a weekend on this, you’re coming with me.
The machine doing the work
My PC isn’t a server farm. It’s my gaming PC: an Intel Core i9-9900K, 32 GB of RAM, and an RTX 5070 with 12 GB of VRAM. That last number is the whole story of local AI. The processor and the RAM matter, but VRAM is the door the model has to fit through. Too big, it doesn’t matter how smart it is. It’s not coming in.
The plan, already decided: a second SSD goes into the machine, dedicated to this. It gets Ubuntu (a free Linux system), completely separate from my Windows drive, which stays exactly as it is for games and work. Two drives, two lives. When I want the AI lab, I boot the Ubuntu drive. When I want to play, I boot Windows and the lab simply does not exist. Clean, reversible, and if I mess up the Linux side in some spectacular way, my games never even hear about it.
The brain: the Qwen family, and the VRAM math
The model family I am going with is Qwen, Alibaba’s open weight lineup, and the current generation is Qwen3. Open weight means the model itself is a file you can download and keep, in this case under an Apache 2.0 license, which is legal speak for “yes, really, it’s yours.” Qwen3 comes in sizes from tiny (0.6 billion parameters) up to a 32 billion parameter dense model, plus bigger mixture of experts versions that are not invited to my house.
Here is the napkin math that decides everything, verified against current local AI hardware guides as of November 2026. A model stored at the community standard 4 bit compression (called Q4_K_M in the GGUF format everyone uses) takes roughly 0.56 GB per billion parameters, plus extra room for its working memory, the context cache. So on my 12 GB card:
- Qwen3 14B at 4 bit: about 9 to 9.6 GB on disk and in VRAM. Fits, with room left for a healthy conversation length. This is my main brain. On a card with the RTX 5070’s memory speed (672 GB per second of GDDR7 bandwidth), a model this size answers fast, faster than reading speed.
- Qwen3 8B at 4 bit: roughly 5 to 6 GB. The quick one. For fast questions, summaries, and anything where the 14B’s extra brains are not needed. Small models also sip power and respond almost instantly.
- Qwen3 32B at 4 bit: about 20 GB. Does not fit. Full stop. It would spill into system RAM and slow down to a sad crawl, and I refuse to build a machine whose main feature is waiting. The 32B is a model for 24 GB cards, and pretending otherwise is how people waste weekends.
That’s the honest headline of local AI in 2026: a 12 GB card is a 14B card, and a good 14B is genuinely impressive. It writes, it summarizes, it explains, it helps with code, it drafts scripts with me. It’s not the smartest AI on Earth. The smartest AI on Earth is a building sized model in the cloud. Mine is a very capable one that lives in a box under my desk and works during a blackout.
The software: Ollama first, LM Studio on the side
Two free tools do this job, and both are built on the same llama.cpp engine underneath, so the speed difference between them is tiny. You pick by personality.
Ollama is the simple one. Install it, type one command, and it downloads the model and starts a conversation. It also runs quietly in the background as a local service, which matters for phase two of my plan: feeding the AI my own files (my scripts, notes and research) so it can answer questions about my own work. That technique is called RAG, retrieval augmented generation, and it’s why a small local model can still be deeply useful: it doesn’t need to have memorized my files, it just needs to search my folders well.
LM Studio is the friendly one, with a proper app window, a model browser, and sliders instead of commands. If Ollama is the garage tool, LM Studio is the one with the nice handle. I’ll probably keep both: LM Studio for trying new models like a kid in a candy store, Ollama as the always on engine.
What it can do, and what it cannot
Let me grade this fairly, because local AI hype is as dishonest as cloud AI marketing.
It can: write and rewrite text privately, summarize long documents, brainstorm video ideas at 2 AM with the internet unplugged, help with code and Linux commands while I am literally setting itself up (my favorite part), answer instantly with zero queue, and never once say “you have reached your limit.” Cost per question after setup: the electricity, which for a chat is pocket change.
It cannot: browse the web or know anything newer than its training, reason at the level of the giant frontier models on genuinely hard problems, generate at their quality on images or video (different tools, heavier appetites), or hold a conversation the size of a novel in its head at once. Long documents need the RAG trick or chunking. And when a shiny new model drops, nobody updates mine for me. I’m the IT department now.
Why bother, then?
Privacy is the big one. My scripts, my notes, my unreleased video plans, my dumb 2 AM questions: none of it leaves the room. Ownership is the second: this AI can’t be repriced, nerfed, renamed, or shut down by a blog post. And the third reason is the one nobody puts on a spec sheet: it is fun. Building a mind into your own PC, on a drive you installed, with parts you chose during a memory price crisis (yes, my 32 GB of RAM is suddenly a luxury asset, see our RAM crisis coverage on this site) is the most PC hobbyist joy I have had in years.
The downsides, printed in the same size font: a weekend of setup, a big model download, Linux moments where a driver and I will have a disagreement, and a permanent quality gap versus the best cloud models. I accept all of it, with eyes open.
The build is not done yet. This is the plan and the reasoning, locked in. When the Ubuntu drive is in and the first local answer scrolls across my screen, part two of this story writes itself, and I’ll tell you exactly how it went, including every stupid mistake. Especially the stupid mistakes. Those are the best part.
Sources
Alibaba Qwen, Qwen3 model family announcement and model cards (dense sizes 0.6B to 32B, Apache 2.0 license, up to 128K context), 2025 to 2026. Ollama model library, Qwen3 download sizes at default Q4_K_M quantization (14B at roughly 9.3 GB), checked November 2026. Local LLM hardware guides and VRAM calculators (Q4_K_M at about 0.56 GB per billion parameters plus context cache, 12 GB cards rated for the 12B to 14B class, Qwen3 32B at about 20 GB requiring 24 GB cards), October to November 2026. NVIDIA, GeForce RTX 5070 specifications (12 GB GDDR7, 672 GB per second memory bandwidth). LM Studio and Ollama official documentation (llama.cpp based local inference, offline operation).
The Checkpoint Poll
Leave a Reply