Down the Agentic AI Rabbit Hole

ganthore — Wed, 27 May 2026 00:00:00 +0000

TL;DR</strong>: I’ve been building out a full agentic AI stack — local GPU compute through Lemonade</a> and ROCm</a>, heavily customized OpenCode</a> and Claude Code</a> setups, and a web-style agent called Gaia</a> running on local hardware. It’s excessive. I love it.</p>

Section</th>

Summary</th></tr></thead>

Local Compute: Lemonade + ROCm</a></td>

GPU-accelerated local inference on AMD hardware</td></tr>

OpenCode</a></td>

Wiring up a local-model coding agent with MCP</td></tr>

Gaia</a></td>

AMD’s local agent platform, packaged for Arch</td></tr>

Claude Code</a></td>

What I’ve been building on top of it</td></tr> </tbody></table>

At some point I went from “casually interested in AI tooling” to “I have an Ansible playbook managing a local model farm and I’m weirdly proud of it.” Hard to say exactly when that happened. Doesn’t matter. Here’s what I’ve been building.</p>

Local Compute: Lemonade + ROCm
The foundation is Lemonade</a>, an AMD project that serves LLMs locally with full GPU acceleration through ROCm</a> and Vulkan</a>. It runs an OpenAI-compatible API locally, which means anything that talks to OpenAI can be pointed at your own hardware instead. Under the hood it’s llama.cpp</a> doing the actual inference work.</p>
The machine behind all of this is a Ryzen 9 7950X3D</a> paired with a Radeon RX 9070 XT</a>. The 7950X3D handles CPU inference and anything that doesn’t need the GPU; the 9070 XT takes everything ROCm can throw at it. One wrinkle: the board also has an integrated GPU, and ROCm’s idea of a good time is apparently enumerating both and then crashing. A one-line config isolating the dGPU fixed it, but it’s the kind of thing you only find out about after you’ve already stared at a crash log for longer than you’d like to admit.</p>
Getting ROCm working on Arch Linux</a> is the kind of experience that makes you briefly consider a career in farming, but once it finally clicked — aye god — I had a local inference server actually using my GPU instead of farming the work out to a data center somewhere in Virginia.</p>
The current roster of models I’m actually using:</p>

Qwen3.5-9B</a></strong> — the main workhorse, solid general-purpose performance at a size that doesn’t require a prayer before each inference</li>
Qwen3.5-4B</a></strong> (via vLLM) — when you want Qwen but faster and cheaper</li>
Gemma 4 4B</a></strong> — Google’s efficient 4B, punches above its weight for general tasks</li>
SmolLM3-3B</a></strong> — HuggingFace’s 3B model, absurdly capable for its size</li>
Z-Image-Turbo</a></strong> — Alibaba TongYi’s image generation model, currently sitting at #1 open-source on the text-to-image leaderboard; yes I’m generating images locally too</li> </ul>
Laid out as a table, because that’s the kind of thing a table is for:</p>
Model</th> Size</th> Backend</th> Where it earns its keep</th></tr></thead>

Qwen3.5-9B</a></td> 9B</td> llama.cpp</td> The workhorse — general-purpose, big enough to trust on most things</td></tr>
Qwen3.5-4B</a></td> 4B</td> vLLM</td> Qwen quality at higher throughput when latency matters</td></tr>
Gemma 4 4B</a></td> 4B</td> llama.cpp</td> Punches above its weight on general tasks</td></tr>
SmolLM3-3B</a></td> 3B</td> llama.cpp</td> Absurdly capable for 3B — the small-and-fast slot</td></tr>
Z-Image-Turbo</a></td> —</td> diffusion</td> Local text-to-image</td></tr> </tbody></table>
The whole thing is managed as a Podman quadlet</a> and deployed via Ansible, because consistency matters and I’ve already written about my feelings on running containers the right way.</p>
OpenCode
OpenCode</a> is my primary coding agent when I’m not using Claude Code, and I’ve been building it out — all pointing at Lemonade as the backend. I’ve got local models set up for different task sizes, language server support for the things I actually write, and a growing list of MCP servers handling everything from documentation to tooling integration.</p>
The MCP side of things has gotten a bit out of hand in the best possible way. I’ve connected it to several tools and services I use regularly, which has turned a code editor into something that can reach well beyond just writing code.</p>
I also have my personal knowledge base hooked into the MCP ecosystem, which gives agents access to notes, project context, and references. Both OpenCode and Claude Code draw from the same memory layer. Agents forget things between sessions by default; pointing them at shared notes is one way to patch the hole.</p>
No API costs, no rate limits, nothing leaving the machine. That’s the pitch and it holds up. The catch is that local models aren’t Claude — they’re better than you’d have believed two years ago, but they have a ceiling and I’m not going to pretend otherwise.</p>
Gaia
Gaia</a> is AMD’s local AI agent platform — essentially a web-style interface for running agents on your own hardware. I packaged it for Arch (gaia-amd on AUR</a>) and pointed it at Lemonade. It’s less CLI, more browsable — a different way to interact with the local model stack. Still figuring out where it slots in, but it’s interesting enough to keep around.</p>
Claude Code #</span></a> </h2>
Claude Code is the other half of the daily setup. I’ve been extending it heavily — tooling for session continuity, workflow automation, multi-agent coordination, and making agent instructions portable across tools. The goal throughout has been making it feel less like a one-shot assistant and more like something that actually belongs to your workflow — persistent context, repeatable patterns, agent instructions portable enough to take anywhere.</p>

Local AI is actually useful now — not “useful if you squint” but useful for real work. The models are good, the tooling has matured, and the ROCm situation on AMD hardware is dramatically better than it was a year ago.</p>
For the hard stuff, Claude is still carrying it. The local models earn their keep on the stuff where keeping things on-prem matters more than raw capability — background jobs, quick questions, anything you’d rather not pipe through someone else’s API.</p>
That gap is closing though. Slowly, but it’s moving.</p>

Open Questions #</span></a> </h2>
Things I don’t have answers to yet. Where does Gaia actually fit — is a browsable web-style agent worth keeping once the CLI tooling does most of what I need, or is it solving a problem I don’t have? The local-vs-Claude gap is closing, but is it closing because local models are getting better or because I’m getting better at picking tasks they can already do? And the shared memory layer between OpenCode and Claude Code — at what point does “persistent context” become a liability, where the agent confidently remembers something that stopped being true three weeks ago?</p>
I’ve been in tech since I was five years old. I’ve watched enough “this changes everything” moments to know which ones to ignore. This isn’t one of those.</p>

fosstog - local-ai

Down the Agentic AI Rabbit Hole