The Ugly Truth About Self-Hosting AI Agents

When I started building AI agents, I thought self-hosting was the obvious move. I already had a server with two GPUs. Why pay cloud prices when I had the hardware sitting right there?

Six agents later, I have a more honest take. Self-hosting some things makes sense. Self-hosting everything is a way to lose sleep and ship slower. Here's what I actually learned.

What I'm Self-Hosting

I run six agents across two machines — an RTX 3090 and an RTX 2060, both in my office. Here's the breakdown:

• Clara (AI receptionist): Voice pipeline runs locally — Whisper for STT, Kokoro for TTS. The LLM runs on Groq because I can't beat their latency locally.
• ResumeForge: Frontend on Vercel, LLM via API. Nothing self-hosted except the ATS parser.
• AgentSeek: Postgres + pgvector on my server. API on the same box. Frontend on Vercel.
• Agent Monitor: Runs locally. Needs to — it's watching the other agents.
• Local-Eye: Fully local. News pipeline + LLM inference on the 3090.
• Shopify brain: Ollama model running locally for product Q&A.

So it's a mix. Some things make sense to self-host. Some things I self-host because I'm stubborn. Let me be honest about which is which.

The Real Cost

People talk about self-hosting like it's free. It's not.

The hardware cost me about $2,400 total — the 3090 was already in my desktop, the 2060 I added later. A 477GB USB SSD for model storage was $60. The NVMe boot drive was already there. So the sunk cost argument is real — I already had most of it.

But the ongoing costs aren't zero:

• Electricity: Two GPUs under load pull about 350W. At my local rate, that's maybe $30-40/month if I'm running inference constantly. Less if the GPUs are idle.
• Internet: I need a reliable upload connection. I'm on Tailscale for access, which is free, but if my ISP has an outage, everything goes down.
• Time: This is the one nobody counts. I spend maybe 3-5 hours a week on maintenance — updates, debugging, monitoring. That's time I'm not building.

The electricity is cheap. The time is expensive.

What Breaks at 3am

Here's a list of things that have actually gone wrong in the last three months:

• Ollama crashed and took down the Shopify brain. Agent Monitor caught it, but the model had to reload — 90 seconds of downtime.
• My ISP did maintenance at 2am and the connection dropped. Clara couldn't reach Twilio. Calls went to voicemail.
• The USB SSD unmounted itself randomly. Half my models disappeared. Took me an hour to figure out why nothing was loading.
• Postgres ran out of disk space because AgentSeek's logs weren't rotating. The whole registry went read-only.
• A GPU driver update broke CUDA. Spent a Saturday morning rolling back drivers instead of building.
• Power flickered during a storm. No UPS. Everything rebooted, but Ollama didn't auto-start because I forgot to enable the service.

None of these are exotic failures. They're mundane. And every single one took time to fix — time I wasn't spending on the actual product.

What I Moved to the Cloud

After the third 3am wake-up, I started being honest about what needs to be local and what doesn't.

Moved to cloud:

• LLM inference for Clara — Groq's llama-3.3-70b is faster than anything I can run locally. Sub-100ms time-to-first-token. My 3090 can't match that.
• AgentSeek's frontend — Vercel free tier. No reason to host a static site myself.
• ResumeForge's LLM — API calls are cheap enough, and the latency is better than local.

Kept local:

• Whisper STT for Clara — runs fine on the 2060, saves me API costs.
• Kokoro TTS — local voice generation, no per-character API fees.
• Postgres for AgentSeek — I need full control over the database, and pgvector works better when I can tune it.
• Agent Monitor — it's watching the other agents, so it needs to be on the same network for low-latency heartbeats.
• Local-Eye's news pipeline — batch processing, runs overnight, doesn't need to be fast.

The pattern: anything that needs to be fast goes to the cloud. Anything that needs control or runs in batch stays local.

The UPS Problem

I don't have a UPS. I know I need one. Every storm season, I'm gambling.

A decent UPS for my setup would cost maybe $150-200. I keep not buying one because it feels like a boring purchase. But the cost of a power flicker — corrupted databases, models that don't reload, services that don't restart — is way higher than $200 in wasted time.

I'm writing this section as much for me as for you. I need to buy a UPS.

Monitoring Is Non-Negotiable

The single best decision I made was building Agent Monitor before building more agents. Every agent registers and sends heartbeats. When something goes down, I get an alert — not a user complaint.

Before Agent Monitor, I'd find out an agent was down because something broke in a way I noticed hours later. Now I know within 30 seconds.

If you're self-hosting, you need monitoring. Not a fancy observability stack — just something that pings each agent and tells you when it stops responding. Agent Monitor is maybe 500 lines of Python. It's the most valuable code I've written.

The Honest Verdict

Self-hosting AI agents is worth it if:

• You already have the hardware
• You're technical enough to maintain it
• You're okay with 3-5 hours a week on maintenance
• Your use case needs control that cloud APIs don't give you

It's not worth it if:

• You're spending more time on infrastructure than on your product
• You don't have monitoring
• Your uptime requirements are higher than your ISP's reliability
• You're doing it for the principle instead of the practicality

I was in that last category for a while. "I should self-host because it's the right way." Honestly, that's ideology, not engineering. The right way is whatever keeps your agents running and lets you spend time building instead of fixing.

For me, right now, that's a mix. Local where it matters, cloud where it's faster. The split will probably keep changing as my needs do. The lesson isn't "self-host" or "use cloud" — it's to be honest about which one each component actually needs.

← Back to blog