The original post: /r/datahoarder by /u/tapdancingwhale on 2024-09-24 00:54:46.
I've browsed through the posts on this sub and r/datacurator and haven't found the right software that fits my needs.
I have a daily-growing collection, about 200TiB of data right now, spanned across LTO-4 tapes and HDDs. I want to organize/manage/deduplicate/keep track of all of this data (which consists of YouTube rips, Linux isos, game ROMs, photographs, music, movies, TV shows, various tarballs, website archives, ebooks, git repos, etc) and neatly have it accessible via tags. So instead of putting firefox-addon-0.3.6.xpi inside of /root/software/web_browsers/mozilla/firefox/platform-independent/addons/unsigned/xpi/firefox-addon/0.3.6/compiled/ directory, I can keep the file inside /root/software/firefox-xpi/, log it in this piece of software and apply all of my tags to it. I can then search for whatever tags in this software and it'll show everything with matching tags.
Hydrus Network sounds interesting but seems geared toward photos specifically, and crams everything (files too) inside its own database. I can't have that as I also seed a lot of this data too.
fs-viewer sounded interesting too but the idea of adding tags to files as xattrs sounds risky, as in, some kind of transfers between file systems or pipelines could easily lead to tags getting lost.
GNU/Linux support is a must, as well as being FOSS. It would be nice if it used some kind of database magic under the hood (like PostgreSQL or sqlite3) but this isn't a requirement. Mounting a virtual file system with all of my logged files visible at once would be the best, but isn't a requirement either (remember these are all spanned across tapes and disks. I use no RAID or LTFS). Recording of file hashes would be extremely helpful in cases where I need to verify the contents of a tape or disk, and I can just use this program to check X disk/tape's file integrity. Automatic tag application (possibly through the use of a hopper directory) would be very helpful, as I'm working with millions of files here. Backup-status for each logged file (would show how many copies exist on separate media, and which particular media) would be helpful too. CLI/TUI or GUI softwares are both okay, but being CLI sounds more scriptable. Web interface viewer/manager optional.
Does a software exist that does these things? I've been researching for years trying to find something that does what I've looking for and come up dry each time. I'm not a programmer but have tried making various management tools, all of which never fully come to fruition and are 1/20th baked (far worse than half-baked, trust me).
Any suggestions or advice would be highly appreciated.