🛰️ Won the Power Outage Battle, Lost to a Ping

📆 2025-08-04 16:40

They said running a homelab was a fun hobby. They didn’t tell me it would turn into a full-time job involving power engineering, DNS sorcery, and the occasional battle with an ancient USB cable. But here we are.

⚡ NUT + UPS + Docker = Apocalypse-Proof?

My setup looks like this:

• One humble APC UPS (aka The Battery Brick of Destiny)
• A Dockerized NUT (Network UPS Tools) server running on Alpine, because I like pain and tiny images
• NUT clients checking UPS status with upsc, like obedient little penguins
• Logic to gracefully shut down services when the battery dips below "oh no" levels

Extras:

• Ping tests to check if my nodes are alive or just lazy
• Wake-on-LAN packets to yell at lazy nodes
• Gotify for shouting notifications to my phone like a panicked toddler

The idea? If the grid fails, my cluster becomes a polite gentleman: it warns everyone, saves its work, and lies down until the lights return.

And it worked! ... until yesterday. My internet went down for more than 3 hours. (Not that I have a lot of traffic and everybody noticed that my sava.rocks services were down).

🚫 The Unexpected Boss Fight: Internet Outage

Power stayed on. But the internet died.

Suddenly, my proud little cluster — which survived power chaos like a champ — sat idle, confused, alone, unable to resolve even a simple DNS query.

It was like watching a bodybuilder forget how to open a peanut jar.

Why? Because while I protected against power failure, I forgot the other lifeblood of modern homelabs: connectivity.

Without internet:

• My Gotify alerts went nowhere.
• DNS? Nah. Local resolution was so last week.
• My cluster, once mighty, was now just a blinking box of existential dread.

📶 Time for Plan B: A Bucket of Wireless Internet

I needed a backup connection. A plan. A dream.

So, what counts as emergency internet?

• A 4G/LTE modem stuffed into the router like a digital EpiPen
• Tethering from an old Android phone named "HotspotOfLastResort"
• A Raspberry Pi trained to scream "INTERNET DOWN!" via text-to-speech

Okay, maybe not the last one. (Or maybe yes.)

Even better: bake internet monitoring into my NUT logic:

if ! ping -c1 1.1.1.1; then
  echo "Internet down. Releasing the backup modem kraken..." | send-to-gotify.sh
fi

🌐 But Wait — What About the Domain?

Keeping your services online isn't just about having internet. It's about being reachable. So I needed to fix DNS too.

🌀 Option 1: Dynamic DNS Like It’s 2009

• Use DuckDNS or similar to point `my-cool-cluster.duckdns.org` at whichever IP is online
• Set up your router or a script to update the DNS record when the connection flips from "cable" to "we found Wi-Fi in the neighbor's mailbox"
• Bonus: add a cron job that checks if your LTE IP changed, and sighs loudly when it does

🧠 Option 2: Fancy DNS Failover

• Use a DNS provider with health checks like Route 53, Bunny, or Cloudflare
• Set up two A records — one for your main line, one for backup
• If the main IP dies, the provider swaps to LTE like a digital stunt double
• Downsides: sometimes DNS propagation is slow enough to brew coffee between flips

🛠️ Option 3: Self-Hosted DNS With Control Issues

• Run `CoreDNS` or `dnsmasq` in your cluster
• Teach it to serve different IPs depending on ping tests, rituals, or moon phases
• Will it work? Yes. Will it break in hilarious ways? Also yes.

📚 Takeaways From My Internet Faceplant

• ⚡ Power is just one piece of the uptime puzzle.
• 🌍 Internet is the other piece. Sometimes the bigger one.
• 🧩 DNS is the glue holding everything together — or exploding spectacularly.

My cluster lived through a lot of power blackouts like a stoic wizard. But a simple internet outage turned it into a confused puppy.

🔮 What’s Next?

I'm now working on:

• A fully automatic failover to LTE
• Smarter DNS routing that doesn’t require divine intervention
• Possibly duct-taping a Starlink antenna to a drone - if I get a drone :)