Building an (almost) fully self-hosted, sandboxed, agentic software factory

I did 90-95% of the work on this with a local GPU. I still ran Claude and Codex, but they were just writing code to orchestrate API calls against local models, local Whisper, etc. Doing the same with the public cloud would have cost thousands of dollars, instead it cost a few hundred dollars for a frontier lab subscription and tens of dollars for electricity.

I recently got into the DGX Spark and just a month ago, got an AMD Ryzen developer platform. Both are incredibly useful tools for some upcoming projects I'm working on (also related to government corruption), but in my limited experience of trying to run opencode on them, the "good models" are still completely unusable because the system prompt alone requires a minute of thinking.

Still, it's largely replaced the cheap tier of the frontiers that I would otherwise be using. It can be run with older GPUs (a 3090 is ~1k), but the time spent thinking will become a fairly noticeable impediment for staying in the flow.

The next step up would be to run DeepSeek V4 Flash 0731 on two DGX Sparks (~$10k), which serve at 60 TPS and sit somewhere around Opus 4.7 level without 3.8-tier thinking.

However, it is worth noting that, if you are buying this hardware just to serve LLMs, it is not cost-effective. It would take over a decade of continuous use to make back the cost of the DGX Spark setup in 0731 tokens. I'm running this setup because I happen to have a 5090, and the two 3090s in my server were cheap enough when amortised over several years.

[0]: https://github.com/Neroued/ninfer

(Submitted title was "A self hosted AI software factory")

1) I have knowledge DB that I query, spend a lot of time to get "query N models, stream to me all N results, let me pick which one" - only to find out that qwen beat all other contenders (those were picked before I got R9700, so the rest is =<8B parameters)

2) Setup OpenCode to use it and gave it a few tasks:

   - first task (new feature to my MCP) took 40 minutes to complete with zero input from me, while it took 20 for sonnet-5 and sonnet-5 kept bugging me. Results are near identical.
 
   - second task (port a specific version of a package to my flake) it got stuck in a hilarious loop where model already been told what hash to use by nix itself, but it wanted to figure our how to get hash another way for some reason.

   - third task (another task, but much harder than first one with most of the discovery already done), kept doing discovery and running out of context, went through 3 compactions (256k context is what I can fit on R9700). Room got too hot, so I stopped it.

  3) virtual assistant like Hermes but my own: no notes, works great.

The Qwen models are decent, but they don't come close to the full Claude experience I've come to expect. As such, I only use the local models for specific tasks where it makes sense to do so. Really the setup is that my Claude-powered agents are able to incorporate my local model into work it builds out. The agents can perform inference against the Ollama API as they see fit, and I encourage them to do so for tasks where (a) the low-level capacity of the local models make sense and/or (b) where costs can become a concern.

It seems to work well when it comes into play (like having Claude drive a web browsing session but letting Qwen handle much of the actual browser interactions, image analysis, etc.). Still, Qwen just isn't smart enough (or fast enough on my machine) to handle anything agentic that isn't non-trivial.

I don't know most of the stuff there, but at least you can dive there.

Is it perfect now? Absolutely not. Will it continue to get better at a surprising pace? Absolutely yes.

So, just adjust your framing from "we're there" to "we're getting there" and all of your concerns go away.

Before this little experiment I generally used it as a very specific search engine for houses, ‘Find me a house in the country but close to amenities with a workshop barn but close enough on the train to Manchester’. It’s great for this!

I’ve never been on the Hermes subreddit I’m gonna go check it out. I agree with the commenter below, it’s fine when I need no record my reps but don’t really care how. It writes tests but I don’t verify. It’s a manual ‘works or it doesn’t’ Thing. I’d never sell the code to anyone.