xtremeownage

My apologies for the past day or so of downtime.

I had a work conference all of last week. On the last morning around 4am, before I headed back to my timezone, "something" inside of my kubernetes cluster took a dump.

While- I can remotely reboot nodes, and even access them... the scope of what went wrong was far above what I can accomplish remotely via my phone.

After returning home yesterday evening, I started plugging away a bit, and quickly realized.... something was seriously wrong with the cluster. As such, from previous experience, I found it was quicker to just tear it down, rebuild it, and restore from backups. So- I started that process.

However, since, I had not seen my wife in a week, I felt spending some time with her was slightly more important at the time. But- I was able to finish getting everything restored today.

Due, to the issues before, I will be rebuilding some areas of my infrastructure to be slightly more redundant.

Whereas before- I had bare-metal machines running ubuntu, going forward, I will be leveraging proxmox for compute clustering and HA, along with ceph for storage HA.

That being said, sometime soon, I will have ansible playbooks setup to get everything pushed out and running.

Again- My apologies for the downtime. It was completely unexpected, and came out of the blue. I honestly still have no idea what happened.

The best suspicion I have, is disk failure.... and after rebooting the machine, it came back to life?

Regardless, Will work to improve this moving forward. Also- I don't plan on being out of town soon... so, that will help too.

There may be some slight downtime later on as I am working on and moving things around. If- that is the case, it will be short. But- for now- the goal is just restoring my other services and getting back up and running.

Update 2023-07-23 CST

There are still a few kinks being worked out. I have noticed occasionally things are disconnecting still.

Working on ironing out the issues still. Please bear with me.

(This issue appears to be due to a single realtek nic in the cluster... realtek = bad)

Update 9:30pm CST

Well, it has been a "fun" evening. I have been finding issues left and right.

1. A piece of bad fiber cable.
2. The aforementioned server with a realtek NIC which was bringing down the entire cluster.
3. STP/RSTP issues, likely caused by the above two issues.

Still, working and improving...

Update 2023-07-24

Update 9am CST

Working out a few minor kinks still. Finish line is in sight.

Update 5pm CST

Happened to find a SFP+ module which was in the process of dying. Swapped it out with a new one, and... magically, many of the spotty network issues went away.

Have new fiber ordered, will install later this week.

Update 9pm CST

1. Broken/Intermittent SFP+ Module replaced.
2. Server with crappy realtek nic removed. Re-added server with 10G SFP+ connectivity.
3. Clustered servers moved to dedicated switch.
4. New fiber stuff ordered to replace longer-distance (50ft) 10G copper runs.

I am aware of current performance issues. These will start going away as I expand out the cluster. Still focusing on rebuilding everything to a working state.

cross-posted from: https://lemmyonline.com/post/53654

I dug through the prime deals and picked out the relevant devices, for which I have personal experience using, and would recommend to others.

Every device linked, will work with home assistant, more or less natively. The majority of them, are flashable to esphome or tasmota. And ALL of them will work 100% locally.

cross-posted from: https://lemmyonline.com/post/53654

I dug through the prime deals and picked out the relevant devices, for which I have personal experience using, and would recommend to others.

Every device linked, will work with home assistant, more or less natively. The majority of them, are flashable to esphome or tasmota. And ALL of them will work 100% locally.

https://old.lemmyonline.com/

Just, an alternative front-end... if you pretty.... a different front-end.

Apologies for the short outage today, My core switch decided to reboot for unknown reasons.

This thing has been up for close to a year, with no recent changes, or explanations behind the reboot.

cross-posted from: https://lemmyonline.com/post/41230

A real-time sankey diagram showing energy flow within my house.

Details, How-to, etc: https://static.xtremeownage.com/blog/2023/home-assistant-energy-flow-diagram/

Seemed pretty cool.

Do note, I don't control mlym.org, so, wouldn't recommend putting your credentials into it!

Friendly FYI-

Just updated to 0.18.1 rc9 for both UI and FE.

There have been a lot of bugs fixed, and hopefully this should improve the performance quite a bit too.

I picked it up a few days ago, and can honestly say- its quite a bit of fun.

https://store.steampowered.com/app/799600/Cosmoteer_Starship_Architect__Commander/