The original post: /r/datahoarder by /u/Doomed on 2024-09-20 05:56:50.
This only became a problem for me as I've gone through about 5 PCs and 10 hard drives and 1.5 NAS.
I have lots of partial backups stored across many drives. I want to centralize them into one drive and folder structure, then back up the drive using standard methods.
Backup part is easy. The dedupe part is the wild west.
I'm not talking about "similar" or "perceptual" duplicates. That's a rabbit hole of its own with justified complexity and no objective truth. I mean byte exact copies.
I used jdupes back in 2018. Turns out it had a bug and instead of deduping I was de-filing every last copy I had. Noted: dedupe software should be boring, small, and filled to the brim with tests.
I look around. czkawka seems popular. And to be fair, it looks good. To be fair, it doesn't seem to have deleted anything but duplicates since I started running it. But it's GUI based and that introduces all kinds of error sources. It does more than just dedupe. That's great, I want to use some of those extra features. But I don't want that thrown into one program. There should be one tiny program to do this, with plugins or whatever to do all the extra stuff. czkawka has a CLI but it's not well documented. Testimonials for all these programs are uncommon - same with tutorials.
I don't get why this is so hard. It feels like it should be a one line command for a program designed for exactly this. The fclones docs talk about all the things you can do with the software. And one of them is deduplication. But I want the one, time tested, failsafe, dummy proof, dedupe script. This is not something the user should have to write themselves.
fclones is CLI and tops the benchmarks.
The code has been thoroughly tested on Ubuntu Linux 21.10. Other systems like Windows or Mac OS X and other architectures may work.
(Emphasis added). Danger! Danger! Good news though, I can't even find a Windows binary. So you'd have to go out of your way to do something this stupid.
I want a duplicate finder with 10x as many lines of tests as it has lines of code. It should be fail safe. See: https://rmlint.readthedocs.io/en/latest/cautions.html
JDupes cited this, giving me false security: https://github.com/h2oai/jdupes?tab=readme-ov-file#does-jdupes-meet-the-good-practice-when-deleting-duplicates-by-rmlint
I'm even skeptical of command line options. Depending on the setup of the program, you're giving users a loaded gun and telling them to be careful. Something like this design might be safest:
# find the dupes
dupefinder path:\ >found_dupes.txt
# send the dupes we found to the trash
dupetrasher found_dupes.txt
Fclones does look really good. And it uses this design. What triggered the last part of my rant was the "hash" section of the readme. You, dear user, can choose from 1 of 7 hash functions for deduping. When would you ever need this? It adds a surprising amount of complexity to the code for little gain. Deduping in general, and hash selection specifically, is one of those problems where I want Great Minds to tell me the right answer. What's better for hashing in a dedupe context, metro or xxhash3? I don't know, probably xxhash because it's faster but I have no idea. When the hell would a user need a cryptographic hash on their own files for deduping? Why do you think your users can do this calculation on their own?
Globs introduce error. Great! Why not just read from a config file?
Using --match-links together with --symbolic-links is very dangerous. It is easy to end up deleting the only regular file you have, and to be left with a bunch of orphan symbolic links.
Thanks for the heads up, but this shouldn't be possible if it's that dangerous.
After reading through the docs of fclones and elsewhere I'm not even convinced it should operate across folders or drives. There's so much trickery afoot and the risk of failure is so high.
