bOt

joined 2 years ago
MODERATOR OF
 
The original post: /r/datahoarder by /u/CarcajadaArtificial on 2025-07-11 20:03:57.

Hello, I just started using ArchiveBox to store local copies of my bookmarks and articles. Frequently I would store two different pages from the same site that would have repeated images, of course it would be better to not keep this kinds of duplicates. I suppose this is a relatively common concern but couldn't find anything about this in the docs. I also suppose that not all download formats would handle this situation the same way, I was using SingleFile which I suddenly realized that it probably wouldn't be too optimized for this. What would be your recommendation for this?

Thank you

 
The original post: /r/datahoarder by /u/SmokingHensADAN on 2025-07-11 18:14:38.

Hey guys, I need to give a quote for 500x 5TB hard drives. I am already in the process of getting reseller status with a large manufacturer. This is falling into my lap, and I'm in a different industry. What are my options for getting that many hard drives at the best price? I know some, but I would prefer to act like I'm clueless. I am sure Ill learn some stuff..

 
The original post: /r/datahoarder by /u/Even-Mechanic-7182 on 2025-07-11 13:10:32.
 
The original post: /r/datahoarder by /u/manzurfahim on 2025-07-11 09:43:52.
 
The original post: /r/datahoarder by /u/cyrbevos on 2025-07-11 09:40:40.

After 10+ years of data hoarding (currently sitting on ~80TB across multiple systems), had a wake-up call about backup encryption key protection that might interest this community.

The Problem: Most of us encrypt our backup drives - whether it's borg/restic repositories, encrypted external drives, or cloud backups. But we're creating a single point of failure with the encryption keys/passphrases. Lose that key = lose everything. House fire, hardware wallet failure, forgotten password location = decades of collected data gone forever.

Links:

Context: My Data Hoarding Setup

What I'm protecting:

  • 25TB Borg repository (daily backups going back 8 years)
  • 15TB of media archives (family photos/videos, rare documentaries, music)
  • 20TB miscellaneous data hoard (software archives, technical documentation, research papers)
  • 18TB cloud backup encrypted with duplicity
  • Multiple encrypted external drives for offsite storage

The encryption key problem: Each repository is protected by a strong passphrase, but those passphrases were stored in a password manager + written on paper in a fire safe. Single points of failure everywhere.

Mathematical Solution: Shamir's Secret Sharing

Our team built a tool that mathematically splits encryption keys so you need K out of N pieces to reconstruct them, but fewer pieces reveal nothing:

bash
# Split your borg repo passphrase into 5 pieces, need any 3 to recover
fractum encrypt borg-repo-passphrase.txt --threshold 3 --shares 5 --label "borg-main"

# Same for other critical passphrases
fractum encrypt duplicity-key.txt --threshold 3 --shares 5 --label "cloud-backup"

Why this matters for data hoarders:

  • Disaster resilience: House fire destroys your safe + computer, but shares stored with family/friends/bank let you recover
  • No single point of failure: Can't lose access because one storage location fails
  • Inheritance planning: Family can pool shares to access your data collection after you're gone
  • Geographic distribution: Spread shares across different locations/people

Real-World Data Hoarder Scenarios

Scenario 1: The Borg Repository Your 25TB borg repository spans 8 years of incremental backups. Passphrase gets corrupted on your password manager + house fire destroys the paper backup = everything gone.

With secret sharing: Passphrase split across 5 locations (bank safe, family members, cloud storage, work, attorney). Need any 3 to recover. Fire only affects 1-2 locations.

Scenario 2: The Media Archive Decades of family photos/videos on encrypted drives. You forget where you wrote down the LUKS passphrase, main storage fails.

With secret sharing: Drive encryption key split so family members can coordinate recovery even if you're not available.

Scenario 3: The Cloud Backup Your duplicity-encrypted cloud backup protects everything, but the encryption key is only in one place. Lose it = lose access to cloud copies of your entire hoard.

With secret sharing: Cloud backup key distributed so you can always recover, even if primary systems fail.

Implementation for Data Hoarders

What gets protected:

  • Borg/restic repository passphrases
  • LUKS/BitLocker volume keys for archive drives
  • Cloud backup encryption keys (rclone crypt, duplicity, etc.)
  • Password manager master passwords/recovery keys
  • Any other "master keys" that protect your data hoard

Distribution strategy for hoarders:

bash
# Example: 3-of-5 scheme for main backup key
# Share 1: Bank safety deposit box
# Share 2: Parents/family in different state  
# Share 3: Best friend (encrypted USB)
# Share 4: Work safe/locker
# Share 5: Attorney/professional storage

Each share is self-contained - includes the recovery software, so even if GitHub disappears, you can still decrypt your data.

Technical Details

Pure Python implementation:

  • Runs completely offline (air-gapped security)
  • No network dependencies during key operations
  • Cross-platform (Windows/macOS/Linux)
  • Uses industry-standard AES-256-GCM + Shamir's Secret Sharing

Memory protection:

  • Secure deletion of sensitive data from RAM
  • No temporary files containing keys
  • Designed for paranoid security requirements

File support:

  • Protects any file type/size
  • Works with text files containing passphrases
  • Can encrypt entire keyfiles, recovery seeds, etc.

Questions for r/DataHoarder:

  1. Backup strategies: How do you currently protect your backup encryption keys?
  2. Long-term thinking: What's your plan if you're not available and family needs to access archives?
  3. Geographic distribution: Anyone else worry about correlated failures (natural disasters, etc.)?
  4. Other use cases: What other "single point of failure" problems do data hoarders face?

Why I'm Sharing This

Almost lost access to 8 years of borg backups when our main password manager got corrupted and couldn't remember where we'd written the paper backup. Spent a terrifying week trying to recover it.

Realized that as data hoarders, we spend so much effort on redundant storage but often ignore redundant access to that storage. Mathematical secret sharing fixes this gap.

The tool is open source because losing decades of collected data is a problem too important to depend on any company staying in business.

As a sysadmin/SRE who manages backup systems professionally, I've seen too many cases where people lose access to years of data because of encryption key failures. Figured this community would appreciate a solution our team built that addresses the "single point of failure" problem with backup encryption keys.

The Problem: Most of us encrypt our backup drives - whether it's borg/restic repositories, encrypted external drives, or cloud backups. But we're creating a single point of failure with the encryption keys/passphrases. Lose that key = lose everything. House fire, hardware wallet failure, forgotten password location = decades of collected data gone forever.

Links:

Context: What I've Seen in Backup Management

Professional experience with backup failures:

  • Companies losing access to encrypted backup repositories when key custodian leaves
  • Families unable to access deceased relative's encrypted photo/video collections
  • Data recovery scenarios where encryption keys were the missing piece
  • Personal friends who lost decades of digital memories due to forgotten passphrases

Common data hoarder setups I've helped with:

  • Large borg/restic repositories (10-100TB+)
  • Encrypted external drive collections
  • Cloud backup encryption keys (duplicity, rclone crypt)
  • Media archives with LUKS/BitLocker encryption
  • Password manager master passwords protecting everything else

The encryption key problem: Each repository is protected by a strong passphrase, but those passphrases were stored in a password manager + written on paper in a fire safe. Single points of failure everywhere.

Mathematical Solution: Shamir's Secret Sharing

Our team built a tool that mathematically splits encryption keys so you need K out of N pieces to reconstruct them, but fewer pieces reveal nothing:

bash# Split your borg repo passphrase into 5 pieces, need any 3 to recover
fractum encrypt borg-repo-passphrase.txt --threshold 3 --shares 5 --label "borg-main"

# Same for other critical passphrases
fractum encrypt duplicity-key.txt --threshold 3 --shares 5 --label "cloud-backup"

Why this matters for data hoarders:

  • Disaster resilience: House fire destroys your safe + computer, but shares stored with family/friends/bank let you recover
  • No single point of failure: Can't lose access because one storage location fails
  • Inheritance planning: Family can pool shares to access your data collection after you're gone
  • Geographic distribution: Spread shares across different locations/people

Real-World Data Hoarder Scenarios

Scenario 1: The Borg Repository Your 25TB borg repository spans 8 years of incremental backups. Passphrase gets corrupted on your password manager + house fire destroys the paper backup = everything gone.

With secret sharing: Passphrase split across 5 locations (bank safe, family members, cloud storage, work, attorney). Need any 3 to recover. Fire only affects 1-2 locations.

Scenario 2: The Media Archive Decades of family photos/videos on encrypted drives. You forget where you wrote down the LUKS passphrase, main storage fails.

With secret sharing: Drive encryption key split so family members can coordinate recovery even if you're not available.

Scenario 3: The Cloud Backup Your duplicity-encrypted cloud backup protects everything, but the encryption key is only in one place. Lose it = lose access to cloud copies of your entire hoard.

With secret sharing: Cloud backup key distributed so you can always recover, even if primary systems fail.

Imp...


Content cut off. Read original on https://old.reddit.com/r/DataHoarder/comments/1lx2my0/protecting_backup_encryption_keys_for_your_data/

 
The original post: /r/datahoarder by /u/Gamesarefun97 on 2025-07-11 04:29:58.

I have been backing up all of my CD's recently, using fr:ac to convert them to .flac files. I've encountered Enhanced Audio CD's, where they now contain both data and audio tracks. For my dvd collection I have used dvd decrypter to convert all of them into .iso files, as I want to be able to emulate inserting the disk, but I'm not sure how to back up this type of CD. I would prefer for it to end up as something like an iso file, but I haven't been able to find much about the best way to rip these.

Any help is greatly appreciated.

 
The original post: /r/datahoarder by /u/5meohd on 2025-07-11 00:21:45.

Hello!

I am working on a project to combine the collections of myself and a local irl friend. Between the two of us we have over 14,000 discs. Counting for overlapping titles its likely closer to 12,000.

So far I have just been testing PLEX, Make MKV, a 20TB external drive, an old 2015 MBP and various playback devices including my Shield Pro 2019.

We have been successful with ripping and playback of our discs, including UHD discs. We are keeping everything lossless, including supplements, commentaries, etc... Im a videophile with a nice Sony OLED and hes a film geek that actually works in the industry of disc bonus feature production. So between the two of us, we just cant budge on file size. In fact, we are most excited about the project giving us convenient access to compiling various versions and imports of the same film into one folder. So exciting!

My question for you experts -

If Im willing to start with a budget of $2K, can I build something quality that can just be expanded every year as more funds become available? Maybe start with some kind of DIY NAS with 8 bays and PCIe expansion capablities? I havent built a PC since Windows 7 in 2010 and Ive never built a server.

Outside of "youre in over your head, give up", I appreciate any and all thoughts or ideas!!

With gratitude!

 
The original post: /r/datahoarder by /u/Intelligent_Series46 on 2025-07-10 20:27:06.
 
The original post: /r/datahoarder by /u/Phil_Goud on 2025-07-10 15:29:51.

Hi everyone !

Mostly lurker and little data hoarder here

I was fed up with the complexity of Tdarr and other softwares to keep the size of my (legal) videos on check.

So I did that started as a small script but is now a 600 lines, kind of turn-key solution for everyone with basic notions of bash... or and NVIDIA card

You can find it on my Github, it was tested on my 12TB collection of (family) videos so must have patched the most common holes (and if it is not the case, I have timeout fallbacks)

Hope it will be useful to any of you ! No particular licence, do what you want with it :)

https://github.com/PhilGoud/H265-batch-encoder/

(If it is not the good subreddit, please be kind^^)

 
The original post: /r/earthporn by /u/Disastrous_Kiwi_9883 on 2025-07-12 05:50:28.
 
The original post: /r/earthporn by /u/Disastrous_Kiwi_9883 on 2025-07-12 05:37:52.
 
The original post: /r/earthporn by /u/SmellyCat0007 on 2025-07-12 05:10:50.
view more: ‹ prev next ›