Selfhosted

Let's tinker around and accidentally break something.

Have you tried introducing unnecessary complexity?

If it’s stable, it’s not a lab.

That’s infrastructure.

When's the last time you checked if your backup solution works?

logging is probably down

You do, of course have a dedicated rsyslogd server? An isolated system to which logs are sent, so that if someone compromises another one of your systems, they can't wipe traces of that compromise from those systems?

Oh. You don't. Well, that's okay. Not every lab can be complete. That Raspberry Pi over there in the corner isn't actually doing anything, but it's probably happy where it is. You know, being off, not doing anything.

Now try migrating all your docker containers to podman.

You have remote power management set up for the systems in your homelab, right? A server set up that you can reach to power-cycle other servers, so that if they wedge in some unusable state and you can't be physically there, you can still reboot them? A managed/smart PDU or something like that? Something like one of these guys?

Oh. You don't. Well, that's probably okay. I mean, nothing will probably go wrong and render a device in need of being forcibly rebooted when you're physically away from home.

Time to start documenting it!

Have you already tried implementing an identity provider like Authentik, so you can add OIDC and ldap for all your services, while you are the only one that’s using them? 🤔

All of those systems in your homelab...they aren't all pulling down their updates multiple times over your network link, right? You're making use of a network-wide cache? For Debian-family systems, something like Apt-Cacher NG?

Oh. You're not. Well, that's probably okay. I mean, not everyone can have their environment optimized to minimize network traffic.

The rare moment when everything actually works. 😄

The comments in this thread have collectively created thousands of person-hours worth of work for us all...

Don't worry, you're one Docker pull away from having to look up how to manually migrate Postgres databases within running containers!

(Looks at my PaperlessNGX container still down. Still irritated.)

Actually, one thing I want to do is switch from services being on a subdomain to services being on a path.

immich.myserver.com -> myserver.com/immich
jellyfin.myserver.com -> myserver.com/jellyfin

I'm getting tired of having to update DNS records every time I want to add a new service.

I guess the tricky part will be making sure the services support this kind of routing...

You have an intrusion detection system set up, right? A server watching your network's traffic, looking for signs that systems on your network have been compromised, and to warn you? Snort or something like that?

Oh. You don't. Well, that's probably okay. I mean, probably nothing on your network has been compromised. And probably nothing in the future will be.

Never run:

docker compose pull
docker compose down
docker compose up -d

Right before the end of your day. Ask me how I know 😂

I should do some breaking network changes... While tunneled in.

Backups. You're forgetting them.

You can always configure your vim further

All of your systems are set up, but are they capable of being redeployed using a configuration management software package? Ansible or something like that?

Oh. They're not. Well, that's probably okay. I mean, you could probably go manually reproduce configurations, more or less.

You have all your devices attached to a console server with a serial port console set up on the serial port, and if they support accessing the BIOS via a serial console, that enabled so that you can access that remotely, right? Either a dedicated hardware console server, or some server on your network with a multiport serial card or a USB to multiport serial adapter or something like that, right? So that if networking fails on one of those other devices, you can fire up minicom or similar on the serial console server and get into the device and fix whatever's broken?

Oh, you don't. Well, that's probably okay. I mean, you probably won't lose networking on those devices.

Can't believe nobody here mentioned nixOS so far? How about moving all of your configs in a flake and manage all of your systems with it?

Then it turns out your monitoring system failed and FUCK IT'S BEEN A MONTH SINCE THE LAST PROPER BACKUP

You have squid or some other forward http proxy set up to share a cache among all the devices on your network set up to access the Web, to minimize duplicate traffic?

And you have a shared caching DNS server set up locally, something like BIND?

Oh. You don't. Well, that's probably okay. I mean, it probably doesn't matter that your devices are pulling duplicate copies of data down. Not everyone can have a network that minimizes latency and avoids inefficiency across devices.

Do you have a spinning fish display in front of your homelab server, right? We all know the spinning fish improves performance and security, it is a indispensable part of homelabbing

Buy a UPS and setup a NUT server on the spare raspberry pi you have lying around.

heck i really wish we could all throw a party together. part swap, stories swap. show off cool shit for everyone to copy.

help each other fill in the missing pieces

y'all seem like cool peeps meme-ing about shit nobody else gets!

time to test the backups!