Noticeable service degradation on Last Bank, the Wobserver powering The Complete Computer, began on when Matrix Synapse federation latency spiked to multiple minutes and metastasized to a full server outage lasting until .
This event was improperly root-caused and ultimately the multi-day downtime occurred to unforeseen inventory issues, compatibility with un-tested hardware, and improper observability in to the live-system, and improper operational handling of these scenarios.
Background
Last Bank is my New Homelab Build.
4 SATA HDD running zfs in a raidz configuration with 2 replicas of all data
1 SATA SSD running zfs root, VFAT /boot, and a small swap partition
Service degradation
system htop, iotop etc reported that the system was undergoing IO pressure stall, slightly above average load 15, etc.
dmesg was essentially quiet, along with service logs indicating, in general that PostgreSQL on the Wobserver was not able to keep up with the traffic being thrown at it. Turning off some non-critical services helped with service latency to a small extent, but did not resolve the issue.
Multiple non-essential services were shut down over the weekend including mx-puppet-discord which was causing a lot of app service traffic to the Synapse installation, and priority was given to Synapse federation traffic as that was seen to be timing out.
The system is supposed to run a zpool scrub every Monday morning at 02:00, so it was assumed that the IO issues were regular or consistent, and not attached to a scrub operation which is known to consume all available IOPs for the disks.
Using available data, we can guess that in the middle of , the root SSD began to report read errors which were not captured by the system's observability platform, though many alerts for IO utilization were silenced during this time as they fed in to the same platform as untuned alerts for space utilization of the raidz.
//notes.whatthefuck.computer/media/7037c87e71ec4fd56d05ed130afe01fe6abdbf574d0f7b117b1c83179ccd6fdc.png #+CAPTION: a graph of drive IO utilization is presented. at 2am on 01-Jan and 07-Jan you can see the raidz begin to scrub lasting roughly 8-9 hours. The other distinct line is the IO utilization of the SSD. Prior to 03-Jan the SSD would use 50-70% of IO basically serving Postgres and some other services. After this point it was pegged to 100% IO util for roughly 4 days until the next zpool scrub occurred
I posit that on roughly the morning of 03-Jan the root SSD started to return read errors which were not recorded by smartctl_exporter in the observability stack and in fact, smartctl itself reported the drive as healthy, despite registering the errors as "marginal Attributes":
SMART overall-health self-assessment test result: PASSED
Please note the following marginal Attributes:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000e 001 001 006 Old_age Always FAILING_NOW 3873947097
231 Unknown_SSD_Attribute 0x0013 001 001 010 Pre-fail Always FAILING_NOW 169At this point, the understanding was clear that the root device had failed.
Attempted Recovery Steps
The first attempt to remediate this issue was an utter failure and should be investigated in depth.
I removed the disk from my computer and for the first time noticed that it says "refurbished" on the side, and that it was a Seagate 600 SSD manufactured in June of 2013. woops!
On I ordered from Best Buy two 1tb NVMe drives, assuming that I would copy the zpool over, reinitialize /boot and begin to boot off of the two NVMe. In August I purchased a Dual M.2 PCIE Adapter for SATA or PCIE NVMe SSD which would provide the NVMe drives to the system, I originally intended to put this and some other m.2 drives in to my desktop, but did not because the motherboard had two m.2 slots onboard.
I lost this device when moving to a new apartment so when I picked up the NVMe drives from Best Buy and brought them home I was unable to install the drives in the system. At this point, I should have responded by returning both drives to Best Buy and buying SATA SSDs instead.
Rather, I purchased from an Amazon third party seller a different NVMe drive adapter and paid for "overnight" shipping.
The drive adapter was not delivered. I re-crimped cat5e cables for all the items on my rack so that they're all pretty and the right length.
The adapter was delivered at approximately 13:00 whereby I accepted the package.
The server does not boot from NVMe.
The server does not boot from NVMe.
The Supermicro X9DRI is built around Intel's X79 chipset which is too old to boot NVMe devices. Some boards have hacked BIOS/UEFI environments which supported this but doing that when the system was already degraded was viewed as too much of an operational gamble at this point especially when I started finding dead links to firmware which weren't cached in the Internet Archive. This could be Type 2 Fun if it wasn't the core of my computing systems, so.
Even if they can't be boot devices, the OS will support these devices via Linux's PCI drivers, so I can use the NVMe disks as a high-speed DB raidz or SLOG/L2ARC for the HDDs.
So I went back to Best Buy. They had a single 1tb SATA SSD in stock after the holidays, and I bought it.
I jammed my car door shut.
I jammed my car door shut right as it started to snow on me.
My Subaru Outback only has a single key-hole on the entire exterior, in the driver's door. I jammed my seatbelt in the driver's side door with the other doors locked. Two hours and three trips to stores with My Dad and we built a window slimjim out of 6AWG copper wire.
=dd='ing from the pre-prepared NVMe drive to the SATA SSD went smoothly while dinner was made. This disk was installed but the system did not boot by the end of the day.
With an NixOS rescue image customized with my justdoit.nix auto-installer, I was able to log on to the machine, set up a /mnt with all the disks properly mounted which I could nixos-enter in to.
A new system needed to be deployed to the chroot with mount points set to the new partition UUIDs.
In my Nix Grymoire is an invocation to copy a nix closure in to that chroot environment which can be used to push a build to the host.
Grub needed to be reinstalled by building with =NIXOS_INSTALL_BOOTLOADER=1 /nix/store/the-nixos-system-last-bank/bin/switch-to-configuration boot=
had to boot with =zfs_import=1= so that the zpools will import after the rescue environment is closed out.
had to remove the =console=ttyS0= option from the The Wobserver's Configuration which specified
kernelParams[ "console=ttyS0" "boot.shell_on_fail" ];=.
At this point, the machine booted and I re-racked it and am considering this incident resolved, barring future remediation and hardening.
Remediation steps
NEXT investigate smartctl and smartctl_exporter
Really hate how the error counts were reported as 0 throughout this period
NEXT tweak alerting to disregard zpool scrub periods
alerting fatigue is real, even at home.
NEXT tweak raidz pool use alerting
there's still like 1.4TB free, ffs