Running AI Agents at 3am: Uptime Lessons

Here's something nobody tells you about building AI agents: they break at 3am. Not because the code is bad — because everything around them is fragile in ways you don't see until you're running 24/7.

I've been running Clara, my AI receptionist, for a few months now. She takes calls, books appointments, sends SMS follow-ups. During business hours, everything is fine. After hours is where it gets interesting.

The 3am Call That Broke Everything

One night at 3:14am, Clara got a call. The caller asked to book an appointment. Clara tried to reach the calendar API. The calendar API was down — not crashed, just timing out. Clara waited 30 seconds, then the whole call timed out. The caller hung up.

I found out about it the next morning. Not from an alert — from checking the logs manually. That was the first problem.

The second problem: the calendar API had been flaky for 40 minutes before that call. If I'd known, I could have restarted the service or switched to a fallback. But I didn't know, because nothing was watching it.

The third problem: when Clara couldn't reach the calendar, she should have taken a message and told the caller someone would call back. Instead, she just... stopped talking. The error handling wasn't there.

Three failures from one flaky API. That's when I started taking uptime seriously.

What Actually Breaks

After a few months of running agents in production, here's what I've seen break:

• LLM API rate limits. Groq is fast, but if you're not watching your usage, you hit the limit and calls start failing. No retry, no fallback, just silence.
• Twilio webhook timeouts. Twilio gives you 15 seconds to respond. If your LLM takes longer than that, the call drops. No retry, no second chance.
• DNS resolution. I had a service that worked fine for weeks, then started failing because the DNS provider had an outage. The service was up. The DNS was down. Same result to the caller.
• Database locks. SQLite is great for single-process apps until two things try to write at the same time. Then you get a locked database and a hung request.
• Memory leaks. Python processes that run for days will eat memory slowly. Not enough to crash, but enough to slow everything down until latency creeps past your timeout.

None of these are exotic. They're the boring failures that happen when software runs long enough.

What I Changed

After the 3am incident, I made five changes. None of them are clever. All of them are things I should have done from the start.

1. Health checks on every dependency. Every external service Clara talks to — the LLM, the calendar, the SMS provider — gets a health check every 60 seconds. If something is down, I get a Telegram message. Not an email I'll read in the morning. A message that wakes me up.

2. Graceful degradation. If Clara can't reach the calendar, she doesn't freeze. She takes a message, tells the caller someone will call them back, and sends me the details. The call still ends well, just not perfectly.

3. Automatic restarts. I wrapped every agent in systemd with Restart=always. If the process crashes, it comes back. I also added a watchdog that restarts the process if it stops responding to health checks, even if it hasn't crashed. Hung processes are worse than crashed ones.

4. Circuit breakers. If an API fails three times in a row, stop trying for 60 seconds. Don't keep hammering a dead service. The caller gets a "we're having technical difficulties" message, which is better than silence.

5. Structured logging. Every call, every API request, every error — logged as JSON with timestamps. When something breaks at 3am, I can grep the logs and see exactly what happened in what order. Before this, I was reading unstructured text logs and guessing.

The Ugly Truth About Self-Hosting

I run most of my agents on a single server. It's cheaper and I control everything. But "I control everything" also means I'm responsible for everything. When the server reboots for updates at 4am, my agents go down. When the network blips, my agents lose their connections. When a Python library updates and breaks something, I find out at runtime.

Managed platforms — Vercel, Railway, whatever — handle a lot of this for you. They restart your processes, they handle SSL, they give you logs. The trade-off is cost and control. I chose control because I'm cheap and I like knowing exactly what's running. But I spend time on infrastructure that I could spend on the product.

If I were starting over, I'd put Clara on a managed platform and keep the experimental stuff on my own server. The production stuff needs to just work. The experimental stuff needs to be hackable. Different needs, different hosting.

Monitoring That Actually Helps

I built a simple monitoring dashboard — the Agent Monitor — that tracks all my agents in one place. It shows uptime, last heartbeat, error rates, and response times. Every agent sends a heartbeat every 30 seconds. If a heartbeat is missed, I get a Telegram alert.

The key insight: monitoring is useless if it doesn't wake you up. I had email alerts for a while. I read them in the morning. The damage was already done. Telegram messages on my phone with the sound on — those I actually see at 3am.

I also learned to monitor the things that matter, not the things that are easy to measure. CPU usage is easy to graph. It's also useless — my server runs at 40% CPU and that tells me nothing about whether Clara is answering calls. What matters: is the agent responding to health checks? Is the LLM latency under 800ms? Are calls completing without errors? Those are the metrics that tell you if the system is actually working.

What I'd Tell Someone Starting Out

If you're building your first AI agent and planning to run it in production:

1. Add health checks before you add features. If you can't tell when it's broken, you can't fix it.
2. Handle every external API failure. Assume every service you depend on will go down. Have a fallback for each one.
3. Use systemd or something similar. Automatic restarts are not optional for 24/7 services.
4. Log everything as structured JSON. Future you will thank present you when something breaks at 3am.
5. Set up alerts that actually reach you. Email is for mornings. Telegram is for now.

None of this is rocket science. It's just the unglamorous work of keeping things running. The fun part is building the agent. The important part is making sure it's still running next week.

← Back to blog