Data Hoarder

221 readers

1 users here now

We are digital librarians. Among us are represented the various reasons to keep data -- legal requirements, competitive requirements, uncertainty of permanence of cloud services, distaste for transmitting your data externally (e.g. government or corporate espionage), cultural and familial archivists, internet collapse preppers, and people who do it themselves so they're sure it's done right. Everyone has their reasons for curating the data they have decided to keep (either forever or For A Damn Long Time (tm) ). Along the way we have sought out like-minded individuals to exchange strategies, war stories, and cautionary tales of failures.

founded 2 years ago

MODERATORS

communick@selfhosted.forum

Do you recommend compressing, encrypting and/or applying error correction (PAR2) to your backups before uploading them to cloud storage? Why (not)? (alien.top)

submitted 2 years ago by C4Oc@alien.top to c/datahoarder@selfhosted.forum

3 comments fedilink hide all child comments

I'm asking this because I'm not entirely sure if it makes sense to do all that. Encrypting files offers more privacy, but then you have to think about managing the keys (and nonces or IVs) and make sure you don't lose them. Would XChaCha20-Poly1305 be ample for this purpose? Compression also makes sense to make the most of your storage space; from what I've heard, ZSTD should be a pretty good contender. I was thinking about using PAR2 too but there is no crate for it, I'd have to settle for launching a CLI with arguments, make my own crate for it or try another language, or look for another solution to error correction.

Although, is it even worth the time (and effort) to reinvent the wheel? Would PAR2 even be the best choice? And if you were to use encryption and/or compression with PAR2, would you apply PAR2 last? I'm thinking you would, despite ChatGPT saying otherwise. I'd just like to hear some opinions on all this. Before I forget, I should mention I don't have (much) money to spend.

you are viewing a single comment's thread
view the rest of the comments

[–] DataHoardingGoblin@alien.top 1 points 2 years ago

I have some thoughts on this.

Would XChaCha20-Poly1305 be ample for this purpose?

Yes. But there's a saying "Don't roll your own crypto" that applies here. Cryptography is very much something that should only be done by people who know what they're doing. Even the pros screw it up from time to time. I'd encourage you to write your own encryption program for educational purposes, but not to rely on it for real security until you get a lot of crypto experience under your belt.

A lot of people here will recommend rclone. This is a good recommendation, but rclone's crypt remote has a few limitations that you should know about:

It does not obfuscate file sizes or the directory structure and file tree. This could allow an attacker who has access to the ciphertext to identify known collections of files. Don't count on rclone's encryption to help you deny being in possession of a pirated copy of The Simpsons, for instance.
It does not protect your files from a malicious cloud provider that wants to truncate or rearrange your files without being detected.

Rclone is still fine for encrypting photos that you took yourself, documents, etc. It'll keep your private data from being sold on the dark web if your cloud account is hacked and dumped.

I was thinking about using PAR2

PAR2 saved me once when I had important files on a failing hard drive. It probably won't help you much with cloud stuff though since most competent cloud providers have their own error correction. If you do use PAR2, apply it to the ciphertext, not the plaintext. If you get bit rot, you'll need to repair the ciphertext to be able to decrypt it if it was encrypted with authenticated encryption.