The original post: /r/datahoarder by /u/OverqualifiedTech353 on 2024-06-17 22:38:12.
Clear & open your minds - this one's a bit of a doozie...
So I've recently begun ROM collecting (might as well reveal that since my account is tied to it), and I'm faced with the best way to preserve this data, as well as use it infrequently. Currently the plan is sort of like a 2.5-2-0.5 method:
- Leave a copy of smaller ROM sets on my desktop SSD (non-OS), but move everything else:
- AND Offload collections (such as handhelds) to 2 or 4TB 3.5" HDDs depending on the collection size and keep in "lukewarm" storage (plug into HDD dock for small updates, otherwise keep offline) [1]
- AND from there, copy organized folders to accessible micro SD cards for the various handhelds (no power most of the time) [2a]
- XOR copy organized folders to portable HDDs for the various consoles (off/idle, or online for maybe max 1khrs/yr) [2b]
- AND Offload copy of entire collection to a large NAS. [3]
- AND Offload collections (such as handhelds) to 2 or 4TB 3.5" HDDs depending on the collection size and keep in "lukewarm" storage (plug into HDD dock for small updates, otherwise keep offline) [1]
I have 6x 2TB HDDs, and 8x 4TB HDDs inherited from various places for the [1] and [3] tasks. So:
[3] For NAS, I am thinking 4TBx6 (-1) = 20TB in RAID5. This would be used for other files and backups as well (that's it's own problem).
[1] That leaves 2x6TB + 2x4TB = 22TB of curated backup/cold storage. Though I am thinking I leave one or both of the 4TBs for replacement drives in case of failure... so 12TB of 2TB drives.
That's sufficient for 2x copies (one online [3], one offline [1]) + the 'accessible'/usable copies [2] (kind of satisfying the "3" and 2 in 3-2-1). The 1 is a 0.5 since it would be in another room(s), and doing a truely remote location is not really in the picture for me. Really all 3 mediums would be in different rooms, likely.
My biggest concern is maintaining the 2 (well, 3) cold storage media types. Especially the microSDs, which I cannot find a definitive answer on how to maintain file integrity (seems like power on at least every year?, then somehow do a bit-by-bit rewrite?)
I've read up on store type archives, and PAR2 type parity. It seems like the best solution for long-term storage for a standalone, non-RAID drive (SD or HDD), would be to create a 10% parity file for every game (zip multi-file games), particularly on the microSDs which are most prone to bit rot.
I've had a lot of issues with microSDs in the past failing, so I have little faith in them. That means I would be forced to keep a per-card mirror on the NAS most likely (in place of the compressed versions for transfer convenience), and backing up any save games at least annually. But then what? Just wait until I try to access a file that fails? And what about the portable and internal HDDs? How often should I be plugging them back in and doing X?
tl;dr: How do you maintain microSDs and offline HDDs, how do you validate them, how often do you clone and replace them? And otherwise what would you do differently or add?