Core Switch 2 Failure

Robert F.P. Ludwick

August 28, 2026

No Rest For The Weary

Here I sat, thinking that I’ve just undergone a couple of weeks of homelabbing, having upgraded a fair number of software versions. Surely I could let my homelab bake for awhile without intervention. Surely… Surely I was wrong.

I came home a week ago from work, intending to take a short break before picking up my oldest from her summer camp. Upon arriving at my garage, I used Home Assistant via CarPlay (a reason I won’t get a Rivian, as they won’t support CarPlay and do not have Home Assistant support) to run an “open home” scene to open the garage door, disable the alarm, and such. The door did not open.

This isn’t the first time I’ve had a strange CarPlay interaction with HA, so I parked and loaded up HA on the phone proper. Gateway failure. My reverse proxy couldn’t hit HA. That meant the pod wasn’t running in Kubernetes. The last time this happened is when my UPC totally failed. I was silently praying that would not be the case this time.

Upon entering my office, I take a look at the rack. Immediately it’s odd to me. The NAS, the UDMP, and Core Switch 1 are all online and operable; they are also plugged into the new UPC I installed when the last failed. Only Core Switch 2 is offline. And wait, what’s that? I watch it for a moment and see a subset of the Ethernet ports quickly blink an LED on then off. It repeats this pattern on a long loop… Has the switch failed? I tried a different power cable; still won’t power on. I tried a different power receptacle, just in case that port on the UPC is dead; no dice. The switch has failed.

Removing Core Switch 2

The failed switch is a Ubiquiti US-16-150W UniFi Switch. I first acquired this switch in mid-2017, so it was in service roughly 9 years. It was the first prosumer-grade switch I owned (as I documented before!). Not bad, I think. I don’t know the expected lifespan for prosumer-grade equipment. I completely disconnected it from power and the devices from the patch panel to the switch. I had to plan out where these orphaned devices would go in my remaining Core Switch 1 (a Ubiquiti US-24-POE UniFi Switch) and my UDMP… I barely had enough spare ports between the two devices to accommodate everything which was plugged in to CS2. So I start replugging from the patch panel to CS1 and UDMP and discover…

Wait. I need 16 PoE ports on CS1 to support 15 PoE devices. And I have several non-PoE devices plugged in to those ports on CS1. Well, first things first, I need to move those devices to non-PoE ports first. This is all made a little more annoying because I have my port mappings documented in an internal docsite, so I have to make sure that I’m temporarily updating those docs as I go along. Ah well, minor annoyance. I get the non-PoE devices moved and start to plug in the orphaned devices, which are:

Upstairs North AP
Downstairs South AP
All 8 Kubernetes nodes

As I start to move over the APs first, I’ve got the UniFi Network Controller loaded up in my browser. I, thankfully, notice some important information: the total maximum PoE output the switch supports. It’s 95W… The one which died supported 150W. I smell a problem, and quickly my worry is confirmed - I do not have enough PoE capacity on CS1 to support all 10 of the PoE devices which need to plug into it, along with the 5 PoE devices which were already running on CS1. Shit.

PoE Trial & Error

Because devices can use a variable range of PoE up to a maximum, I had to engage in trial and error in order to figure out which devices I could ultimately get onto the switch within capacity. My Kubernetes cluster can reliably run one node down (6 of 7 worker nodes, with the control plane as a 7th host). I could try six worker nodes, but didn’t really want to. And ideally I also want all four APs in my home to be online for optimal WiFi coverage (2 PoE APs were already plugged into CS1). First order of business - try dropping one of my three PoE external cameras (the remaining PoE devices on CS1). It didn’t provide enough PoE capacity to plug the remaining 10 devices.

Unplug the second external camera, still leaving enough exterior and interior coverage in case, for some ungodly reason, now after CS2 dies is when a burglar decides my property is their target. Still too much PoE scheduled for the switch. Regretfully I drop 2 of the APs; I know I likely have enough coverage even running 50% of my installed APs. I was ultimately correct on that assumption, though it took about a half-hour for device rebalancing to fully take effect on the remaining 2 APs. I went into the weekend with this configuration, confident that I wouldn’t go over the PoE limit and could ride this setup until a replacement switch arrived.

Nope. Overnight I got a PoE maximum alert. Disconnected the remaining external PoE camera, leaving me with the full Kubernetes cluster and 2 APs. Thankfully, this held until my replacement switch arrived. And the replacement switch? Another Ubiquiti US-24-POE UniFi Switch, just like CS1.

Installation And Moving Ports

This is the part of this affair which is simultaneously exciting and boring. New switch in, hurray! I can fix the setup and get back to normal! Ah, but tedious work replugging and updating documentation. I unplugged the orphaned devices from CS1 and got my new switch, dubbed Core Switch 3, installed. Plugged it to the port in CS1 where CS2 was connected, adopted it, updated software… and it entirely disconnected from the network controller. No clue what happened. So I hard factory reset, readopted it, and thankfully all was well. Something about the software upgrade on the switch caused it to stay disconnected from the network controller. Minorly alarming, but I haven’t had anything amiss happen since.

I had to pull some longer Ethernet cables out of storage to plug in my devices from the patch panel to CS3. The port design on the new switch did not match the old one, and the PoE ports were not close enough to where I had things in the patch panel to keep short cables in play. I didn’t really mind… until the longer cables started causing me problems in getting the door to fully close and latch. Their longer curves pushed out too far. Cursing ensues.

I ended up ordering more short cables. Why, you ask? I should have been using the shorter cables with CS2 before it died, right? Well, during this whole ordeal I noticed at least one of my Kubernetes hosts registering only 100 FDX with the network controller, not 1000 FDX. I thought maybe it was the cable, and I didn’t have any spares. Mercifully, the additional short cables arrived today. I spent the time and shifted the cables plugged from the devices into the patch panel across, that way I could use the short cables from the panel to CS3. I used all-new short cables just to make sure link negotiation speeds wouldn’t be affected by any bad cables I had from the previous set.

The host still only negotiated 100 FDX. And it was the same host… Something for future Robert to deal with. 100 FDX is still sufficient, for now.

Everything else, though, is up and running swimmingly. I’m at full capacity once more - all four APs, all eight Kubernetes hosts, all three exterior cameras. Something to keep an eye on though, CS3 is using 75W-80W of PoE power between its 10 devices. If I get any more PoE devices, they’ll have to go to CS1.