Backups Aren't Simple

aafilipovski about 4 hours ago 11 commentsRead Article on filipovski.net

DE version is available. Content is displayed in original English for accuracy.

⚡ Community Insights

Discussion Sentiment

38% Positive

Analyzed from 902 words in the discussion.

Discussion (11 Comments)Read Original on HackerNews

innocent_name•about 1 hour ago

>Motivated by our success in developing this solution, we try to use it to backup the homelab with its 10 Docker containers. But later we find out from logs on the individual machines that backups are failing. The reason being that many Docker containers like to create root-owned files, and if you’re not careful you can create a cronjob running as the default user.

and a privesc also.

cvhc•2 minutes ago

Yeah it feels it can be risky to "make the backup also dump the databases, and give it full filesystem permissions on our Docker volumes", depending on what binaries get the root privileges.

What I do is having a minimal systemd timer under root that only calls restic for backing up files (with some additional systemd seucurity restrictions). App-dependent backup logics, like dumping databases, are done by user/container-level cronjobs separately.

bloomingeek•about 2 hours ago

"encrypted, chunk-level deduplicated, GFS-rotated, point-in-time archived, cloud, 3-2-1 backup solution" is now my newest password, no commas. (Don't tell anyone!)

rileymat2•about 1 hour ago

Change the o in rotated to a 0 and you might have a secure password.

readthenotes1•about 2 hours ago

"Obviously, none of this is worth anything if you don’t actually test restores."

One of my teachers claimed that contrary to common behavior, no one actually valued backups; it was restores that were worth paying for.

gerdesj•about 1 hour ago

Try running a business on that basis - where do the restores come from? 8)

We charge our customers for backup storage by volume and that includes an annual demonstration of recovery. We fire up some or all of their systems on our gear in isolation and show that they are reasonably functional.

Your teacher's comment is riffing on the well trod lines of: "heights don't kill people, it's depths that kill people", which is all about perspective. This is is not related to "guns don't kill people ..." which is about agency.

al_borland•44 minutes ago

It is true that very few people proactively care about backups. It’s like flossing. People know they’re supposed to do it, but in the moment it feels like a waste of time and energy, when there are more important things to do.

When data is lost, or decay takes over, the value can skyrocket. Depending on what is lost, someone may be willing to pay 100x when they would have spent on backup in order to get their data back, this is why data recovery services can basically charge whatever they want.

My assumption is the Venn diagram between people who pass the marshmallow test, and those who proactively backup their systems, has a lot of overlap.

OkayPhysicist•about 1 hour ago

Running a business on that basis would look like the white(ish)-hat version of ransomware. "Oh, your data is gone? How much is it really worth to you?"

bitwize•about 1 hour ago

"It's not the fall that kills you, it's the sudden stop at the end."

TheCondor•about 1 hour ago

It is a special sensation when you go to your hand rolled backup "solution" to make some simple recoveries and start finding problems. It is a complex mix of not dread but disgust, sick to your stomach, self-anger, and loss. I can only image what that's like with like all of your children's baby pictures being the lost material. Borg and restic are fantastic, it still seems like some sort of dummy proof mode might be desirable, all the features in the world and they have some some reasonably sane policy or something that includes verifications.

On the flip side of that though, really and truly, I've restored a laptop from a timemachine backup and it came back damn nearly identical and that was magical.

innocent_name•41 minutes ago

>Borg ... fantastic

I needed to temporarily move off 20TB from my NAS and bought a Hetzner storage box.

I wanted to encrypt the data before sending it so i went with the borg ssh mount. Long story short, borg failed a couple of times due to https://github.com/borgbackup/borg/issues/7672, i resumed it per documentation and got 2gb of silently corrupted data. There was no way to do data integrity checks due to broken pipes and there was no option to resume the job from the previous point either. I couldn't afford to go with the s3, backblaze wasn't an option due to placing trust into their proprietary client doing the encryption and i got majorly screwed in the end.

PunchyHamster•about 2 hours ago

So is apparently making readable site.

Software like https://restic.net/ does a good job. Few choices of backend

Accepts stuff from pipe too so you can just pipe mysqldump or pg_dumpall without intermediate files

Very decent options for checking repo integrity, personally (well, at work too, we use it on few hundred user machines and servers) I also added "roll a dice for backup and try to restore it" test script to make sure it is working

Decent deduplication too so "store year's worth of weekly snapshot" is very sensible strategy once you exclude the "constantly changing and useless" (caches etc.) files out of it

can mount FUSE directory with all backups on sensible OSes.

https://kopia.io/ does the same +GUI but a bit worse on CLI front (it insists on keeping local config which makes scripting a bit more involved, but not much)

LilBytes•about 1 hour ago

I'm using Restic & Backrest (https://garethgeorge.github.io/backrest/) everywhere I can, they're both exceptional OSS.

IMHO, the easiest way to perform a restore test is to use your production backups to refresh a lesser environment on a weekly cadence. Naturally this has to be appropriate to the type of data you're restoring, e.g., our E2E (end to end) testing environment has been built to the same risk tolerances of production so it's suitable for production data and the PII (Personally Identifiable Information) it hosts. If this isn't in your own risk tolerances, you can still perform the same test but ensure the data from prod is anonymised or alternatively destroyed, and perform a second restore after production restore with the dataset that preceeded it that has no PII data or similar.

With this the E2E tests confirm the backups are expected and you can tick a box that when shit does eventually hit the fan, data corruption isn't a problem. Normal caveats apply, you must make sure your backups are immutable so they cannot be modified or changed _after_ the tests have been ran.

bt1a•about 1 hour ago

importantly, does your die have a side that corresponds to a multi-continental cataclysmic natural disaster of magnitude only known unknown to pre-recorded-history

pizzafeelsright•about 1 hour ago

I don't want backups. I want synced duplicated data in geographically separated by shared credentials.

offmycloud•43 minutes ago

> I don't want backups. I want synced duplicated data in geographically separated by shared credentials.

If you accidentally delete something, how long does it take for the delete to propagate to all of your geographically separated datacenters? You need some sort of point-in-time snapshots to be able to recover from accidents and silent corruption.

layer8•18 minutes ago

Sounds like you are the second type of people mentioned in the article.