The original post: /r/datahoarder by /u/Servletless on 2024-08-22 15:39:23.
I recently had a near-death experience (don't worry I'm safe, but some of my data nearly died) with 2-2-1 backups. My largest data HDD suffered partition table corruption (user error) during an upgrade, and all I had was cloud backups.
I quickly learned a few things,
- cloud backups run with a time delay and prioritize small files first, so newer large files were missing from the cloud backup,
- some tiny percentage of my oldest files suffered a persistent failure to restore: after working with Tech Support of the cloud backup provider who were unable to resolve the failure, the best explanation is that they suffered bit rot in the cloud
- crash recovery software can be effective but can also be "too effective": it recovered 3x the amount of data on the crashed drive, most of it duplicates, corrupt duplicates, and duplicate contents with Lost File Name. This was a massive time-consuming pain to cleanup, even after arming myself with deduplication software.
I've now instituted an emergency 3-2-1 configuration, using ROBOCOPY to make a local backup to an external HDD, while keeping the same cloud provider for now (they gave me a refund and a free renewal to keep me around). I'll eventually replace the could provider, but for now I'm focusing on improving the local component.
I'd like to improve this a bit without making things too complicated. A couple of things bother me about ROBOCOPY:
- The external HDD has no mechanism to detect bit rot (not until you try to retrieve a rotten sector years later). There are no visible checksums anywhere. And as I learned earlier, neither has my cloud provider. So I'd like to incorporate some sort of checksum (MD5, SHA1, etc) integrity check into this whole system.
- The files on the external HDD are not discoverable and not tamper-proof. It's not like you can look at the contents of an old drive and know "oh, this is a full backup from such and such date, and here's a checksum to verify that the entire back-up is intact". There is no way to detect missing files.
- I feel the files are too "exposed" and can be easily tampered with, accidentally or maliciously.
What are y'all using for you local copy in your 3-2-1 setup?