this post was submitted on 02 Nov 2023
1 points (100.0% liked)

Data Hoarder

221 readers
1 users here now

We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time (tm) ). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.

founded 2 years ago
MODERATORS
 

On Linux, I use RAID mirror (mdadm - software reaid). How to configure it so that data is read from both underlying devices and compared, to detect data bitrot and fail in such case?

Is that at all supported?

Would be good option to have.

(Of course as one of layers of protection, for important data)

top 5 comments
sorted by: hot top controversial new old
[–] dr100@alien.top 1 points 2 years ago

Traditionally RAID is for uptime or speed, this isn't that (actually quite the opposite). If you want to do an extra integrity check one of the popular ways to do it is to use a file system that has checksums, like zfs or btrfs.

[–] Party_9001@alien.top 1 points 2 years ago

How to configure it so that data is read from both underlying devices and compared, to detect data bitrot and fail in such case?

You don't lol

Is that at all supported?

Not afaik.

Would be good option to have.

If it ain't broke don't fix it. Use ZFS or BTRFS

[–] Hakkin@alien.top 1 points 2 years ago (1 children)

I believe you can get this if you setup the underlying disks with dm-integrity, though I'm not sure it's really worth it since it's a very manual setup and has lots of potential edge cases and performance issues. Basically when the underlying disks are setup with dm-integrity, each read is verified with a checksum, if the checksum doesn't match, the dm-integrity layer will return a read error and mdadm will log an error and automatically fallback to the secondary mirror. A "scrub" in this case would simply be reading the disk to /dev/null with dd, then checking the mdadm logs for any errors.

Note that it's been a long time since I've messed with this, so it's possible I got the exact details of the setup wrong.

[–] snatch1e@alien.top 1 points 2 years ago

Unfortunately, it's not supported in mdadm. As an alternatives, you may look into zfs and btrfs. Also, backups with versioning may help you if bit rot happened to restore a good version of your file.