The death and rebirth of my home server

Flash memory generally doesn't have great lifespan for repeated writes; using it as a root device is going to kill it pretty quickly. I had a similar problem once running a NAS off a USB stick; within a year of so the stick would no longer accept new data and I swapped it out for an SSD instead.

"Raspberry Pis are notorious for SD card corruption with unknown causes."

This is not specific to RPis. When an SD card is subjected to "unpredictable" writes it creates potential risk of corruption. The writes are triggered by software, not the RPi hardware

There is no rule that says the RPi user must mount the root filesystem r/w on the card. It can be mounted r/w on mfs or tmpfs, for example, and the card can be removed after boot. Been doing this since 2012

Zero writes to the card, no corruption

It's true these corruption issues are "notorious" but that's due to RPi owner behaviour, not the RPI hardware

Alternatively cheapest fix is an m10 optane and nvme enclosure. You maybe need to enable usb quirks in the kernel depending on on enclosure. Costs about 20 bucks all in

That’ll keep the rasp online for essentially forever cause optane isn’t going to wear out

I suspect it isn’t the hardware but the bullheaded approach of running a desktop Linux environment from an SD card.

I have a rasperry pi 2 and 4 that have been running on the same size uSD (reputable brand) for 5 years and haven’t had any issues. I have swap turned off. You just have to be aware of the i/o and not do anything stupid. Clearly for heavy i/o with multitasking they are not ideal.

You can buy cards that support SMART if you want to track wear, but they are expensive.

I’ve got an rpi4 booting from an usb ssd and bought a n100 minipc anyway… last February for $150. Should have bought a 40ft container of these

> Raspberry Pis are notorious for SD card corruption

True SD cards are less than ideal.

But also I suspect half the problem specifically with Pi and SD cards is that people use cheap-ass SD cards and maybe ones they found in the bottom of a drawer that may or may not have previously been used in another device (e.g. camera).

I suspect if people bought industrial SD cards instead of consumer-grade junk they might get a better lifespan out of them.

Make the sdcard a read only root fs, problem solved

My issues with Pis and SD cards ended when I switched to using SSDs as boot drive.

I'm super happy with an Odroid H3 as a NAS: low power usage, silent, x86, nvme + 2 sata ports. Runs home assistant, smb shares and a few more things

Can confirm, I've got a pi 5 here that's been running 24/7 for over a year (possibly two)... it boots fine, but anything else makes it run so slowly because the sdcard is trashed.

The current setup writes to the systemd journal at least twice every minute... It's next iteration is getting a proper nvme.

Just have the raspberry pi boot from a USB/SSD.

Rpi5 has a pcie slot if you want a cleaner setup.

Abusing frontier AI to determine what the hell I did with ancient installs, and then modernizing it, as been quite useful.

There is for hard drives - SMART - the problem is, unless you're an enterprise (and let's all stop and laugh because we KNOW that most "enterprises" run at least some hardware until the only person who even knows it exists is long in the grave) you're not going to do preventative replacement.

And especially for home use, where nien 9s is more realistic than nine 9s, you're better off having a solid backup strategy and a source of spare parts.

My main "tank" server is 16 years old this year, and has blown out one motherboard - it was a bit annoying to be "offline" for a few days as a replacement wandered in from eBay, but it worked. If it happened today I might import the zpool into another device temporarily - or permanently, who knows? This R510 probably takes more power than a datacenter.

My primary home zfs server is from 2010, running 24x7 since then. It has a 4-way mirror zfs pool for data. Over the years I have replaced two of the four drives, so two are still original from 2010. Everything else on the machine is from 2010.

You can get unlucky and have components fail quickly, but usually if they last, they'll last for a long time.

But trying to run a server from an SD card like in the article, seems silly. False economy, just get a good SSD.

For drive health monitoring there is SMART, but IME while it does monitor drive usage and lifetime, it rarely predicts catastrophic failure since that happens suddenly. Both drives that failed in this system went from healthy (according to SMART) to unresponsive overnight.

I'm running my own home server on an old 2014 Mac mini, 24/7 basically. It both has a SSD and a HDD (old "Fusion" disk type of machine, but I use both parts separately, the SSD part for the OS, the HDD part for the data.)

A few days ago the HDD had its 100,000 Power_On_Hours anniversary. :) According to SMART this is way past the expected/ designed life time of the disk (the normalised value reports "1" since a few years when the disk hit about 42,000 Power_On_Hours)

So you never know. It either works or it doesn't. You have backups for the worst case.

I run a fairly large homelab (probably ~10 nodes when including OPNSense routers, backup NAS at our cottage, etc.), and hardware failures have been incredibly rare, especially compared to what it used to be a couple of decades ago (I had quite a few motherboards fail due to blown capacitors due to the stolen electrolyte formula scandal).

Hardware failures tend to follow a bathtub curve: there are more frequent failures early on due to manufacturing defects, then they become rare until you reach the end of the natural lifespan of the equipment. That is shorter for some components (spinning rust HDDs, fans, CMOS batteries, power supplies in noisy environments), but extremely long for others.

> So if that were the case then why isn't there some sore of daemon or system software service that tracks the expected life of the hardware?

Some linux distribution will regularly check SMART data from disk and warn you if some threshold has passed (like number of corrected errors, spin up/down, total hours etc).

That said, if we talk about consumer hardware in a home lab or the like, of the disks that I bought in the past 15 years, only one broke down, and it did fairly quickly so my suggestion is to use redundant storage, make backup and change the disk when it breaks down. (Enterprise disk and usage is another can of worms of course).

Life expectancy really depends on how the hardware gets used. I run a small fleet of servers that are heavily CPU / GPU loaded, and the most frequent hardware failure I see is power supplies, with RAM as #2 and spinning HDDs as #3. Each of these (on enterprise-grade gear) has some amount of reporting / monitoring, but it's rarely the case that I get any advance warning before a failure. The RAM is probably the nicest, as it sometimes starts with correctable ECC warnings. Nearly all of the PSU failures have been sudden deaths, with the only saving grace being N+1 redundant supplies usually keeping the victim server alive.

Hardware wearing out is not common at the individual component level. It only becomes common when you are running a large number of servers, those individual small probabilities add up.

About the only thing that regularly wears out is spinning drives, and those have SMART data to give you an idea of when they'll die, but generally yif you're actually committing to high uptime you'll have redundant hardware so any one server failing doesn't take the service down, giving a bit of a grace period on hardware failures

SSDs have a fairly linear wear-out mechanism although the exact end point is unpredictable as always. HDDs fail randomly or from mechanical shock.

Zram creates a compressed RAM disk. https://wiki.archlinux.org/title/Zram

For anyone trying this with a Pi older than the Pi 4, you might still need an SD card with one file on it: https://www.raspberrypi.com/documentation/computers/raspberr...

You can, I am self-hosting Immich via Docker on Raspberry Pi 4 with small SSD and it's nothing but good experience so far (but I keep the photos backed up elsewhere just in case, I use Immich mostly to share curated sets of photos)

> If i'm reading this correctly, can run Immich from a Raspberry Pi 4? That's pretty cool.

Yes! It runs completely fine on a Pi, except for the machine learning service. The good news is that you can run that service on a separate computer with a GPU and point Immich to it. Immich will use it when it's available and I think you can configure it to run all the queued ML jobs at a certain time when you expect the computer running the ML service to be on.

Personally I would get a used Dell Optiplex / Lenovo ThinkCentre / HP EliteDesk. Either the "mini" / "micro" if you really want small. I am a fan of the SFF (Small Form Factor) size and the Dell Optiplex models. In particular the 70XX versions that offer Intel AMT / vPro for remote management.

I run a Proxmox server on my Dell Optiplex SFF 7070. With the SFF size you can add two low-profile PCIe cards, if desired. In my case I added a 10Gbit NIC.

The reason I would go with a used Dell / Lenovo / HP is that these tend to be well-built machines that run for years and years 24/7 in offices and proven to work well.

I picked Beelink a couple years ago. It didn’t last long. I can’t simply reboot it. I have to unplug it, wait 5 minutes, plug it in, and then boot it. It turns OS installs and even system updates into such a hassle that it’s not even worth using anymore. I wanted to run it headless and just connect remotely, and that doesn’t really work well with this issue, as I need regular physical access.

I assume it’s a bad capacitor or something. I tried changing out the power adapter, but that didn’t help.

I don’t see myself ever buying another Beelink.