It's A Digital Disease!

The original post: /r/datahoarder by /u/blackletum on 2024-06-11 18:15:17.

The original post: /r/datahoarder by /u/Surax98 on 2024-06-11 17:32:13.

Hello guys, I am looking for a raid controller to be used over USB 3.2 10Gbps (I know, it's far from being advisable but that's the only thing I can do right now, since my server is a laptop) and I have to choose between the following:

• IBM M5015 (PCIe 2.0 x8, SAS 6Gbps, 512MB cache DDR2 800MHz) for 20€ and located in my city
• LSI 9271 (PCIe 3.0 x8, SAS 6Gbps, 1GB cache DDR3 idk-how-many MHz) for 33€ shipped from China.

Now, since I am over USB 10Gbps, I am limited in both scenarios, but I'd like to know if that faster and more cache of the 9271 will actually make a difference for a domestic usage. To help you better understand my use cases, I will list them:

• Plex and Jellyfin media server
• IMMICH for photo synching
• Nextcloud for everything else

These are the main I/O bound containers I am running on my server. What do you think? Any other suggestions for a raid controller, instead (which costs less than 40€)?

The original post: /r/datahoarder by /u/69PepperoniPickles69 on 2024-06-11 17:26:46.

Do you know if there's a way to upload my own files so that listen to them instead of reading? Or does it have to include actually uploading books and them getting approved, scripted and so on? And if so do you guys know any alternative website where we can upload large texts for listening with decent quality?

The original post: /r/datahoarder by /u/SlayterDevAgain on 2024-06-11 16:27:17.

I'm about to build a new NAS. With my current NAS I just kind of threw drives at it as I aquired them (2x 2TB in RAID 1, a 4TB and a 6TB drive in pools by themselves, and 5 6TB drives in an external enclosure in RAID 5).

In the new build I have 3 10TB drives I'll be adding. I don't necessarily need to keep all the drives from the old build (I at least want the 5x 6TB drives but not externally) but what would be the best way to migrate the data to the new build? Any advice is appreciated.

Further info: Current build is FreeNAS and I'll probably keep with that or TrueNAS.

The original post: /r/datahoarder by /u/rrybwyb on 2024-06-11 16:16:52.

Original Title: Historical data hoarders at the library of Alexandria lost untolds amount of work and knowledge after the library was burned. Sumerian texts survived 4000+ years due to being written on clay tablets. Is there any efforts to transcribe some of our knowledge into more permanent media?

I mostly hoard books. Its amazing how many I can fit onto just a small 10 tb hard drive. But if that gets wet, dropped, or someone holds a magnet to it, I've lost millions of hours of research and knowledge from 10's of thousands of authors.

Even looking at the history of the dead sea scrolls, Whatever idiots were in charge of transporting those did an awful job. They were taken from the dry desert to a place where they could get humid and rot.

Are there any organizations out there that have transcribed some of our more important items into stone or clay? I've been looking more into Sumerian history and the reason we have many of these items still is because they were carved into clay - and clay can last a pretty long time.

Its kind of short sighted of humans to think we're immune to a giant asteroid or nuclear winter. In that situation, What would humans 5,000 years in the future after a major catastrophe be able to look back on and decipher about the 2000s?

Edit: also I can't believe I forgot about plastic. That supposedly forever product that never breaks down. Does anyone know if it is really as permanent as something like ceramic? I've seen it become quite brittle and disintegrate when left out in the sun for even 6-12 months

The original post: /r/datahoarder by /u/MagicPracticalFlame on 2024-06-11 15:23:59.

I've got a small NAS with 4 x 6tb Drives in. The drives have just ticked over the 40,000 (grim dark) power-on time. Given that all the drives where purchased and installed at the same time, I'm worried about one crapping out and causing a domino effect when I get the replacement in to rebuild the array.

The drives are Seagate ST6000VN0033 drives, health status shows as good on all counts. Just wondering when you guys start to consider replacement or migration to a new NAS (which is what I'd probably do).

The original post: /r/datahoarder by /u/1000Zebras on 2024-06-11 14:54:35.

Hi, 

I'm curious what you guys would implement in this situation in order to, above all, simply maintain at the very least one solid, off-site backup of all of my data files and also, in the event of something happening to my main data drive on-site, reduce downtime as much as possible. 

Here is my current setup as is relevant to the question at hand: 

• OrangePi 5 Plus running dietpi (so pretty much just debian) as my main server on-site
• One eMMC boot drive on the OrangePi containing the OS and all of my docker-compose files, as well as the OS itself
• Recently acquired 14TB external USB drive that houses purely my data for all of my docker containers (and then some outside of those, as well, but not much)
• OrangePi is running Tailscale
• A second RPi that lives at my brother's house also running Tailscale (so any connection between the two will more than likely be running over the interwebs, but through Tailscale) and with a second 14tb drive identical to the other connected to it, ready for data storage

What I'm wondering is what may be the best strategy for maintaining a backup of the main data drive on the secondary drive, ideally in a mirrored fashion such that were the main drive to fail, I'd simply be able to plug in the secondary drive to the OrangePi, mount it at the same mountpoint as primary would have been, and I'd be back up and running nearly immediately (once the drive was physically moved between locations, of course). 

It's worth noting that, at present, I am dealing with nearly 4.5TB of data on main data drive (also currently backed up to the cloud via Kopia and iDrive E2)

I've been considering: 

• Trying out lsyncd or DRDB in order to literally have the drives mirror each other in as near realtime as the connection will allow. I have not used either of these tools yet, however, so I'm not familiar with exactly how they work behind the scene. And also, I realize that it is a lot of data to keep in sync over an internet connection, especially at file or block-level granularity as I believe those tools are designed for. In "normal" usage, I am not necessarily adding or changing all that much data on a day to day basis, but were I to make any major shifts in organization, or simply to add a lot more data into the mix suddenly, I'm wondering if the tools would be able to keep up
• Running an rsync job over ssh at a specified interval (say, maybe, a couple of times a day) in order to keep the two up date. I would of course again run into the same problem that would arise with the first option were I to make any drastic changes, but theoretically I'd eventually always have a 1 to 1 sync/backup between the two drives
• Simply running some sort of backup program from the main Orangepi data drive to the RPi's data drive, again at whatever specified interval (say, maybe, daily). I'd probably have to run some sort of webDAV server on the secondary RPi in order to facilitate backups between the two were I to use Kopia. Or, I suppose I could even run the data drive on RPi on a minio instance and have Kopia backup via the S3 protocol, but this seems perhaps like a little bit of overkill, and it wouldn't necessarily be the sort of 1 to 1 sync I'm shooting for as Kopia would organize the backup data in a fashion that it understands. This would be acceptable, though, as again at the end of the day the most important thing is to have all of the data itself stored safely in both locations, one way or another.

How would you guys go about keeping things in sync between the two data drives? Or, should I just eschew that idea given the limitations of the bandwidth/connection between the two and go for straight backups using Kopia, or some othe rbackup system? 

Please, if you have any thoughts on how you'd architect this scenario, I'd very much appreciate any and perspectives/insights. 

Hopefully that all makes sense. If you need anything clarified, by all means speak up and I'll do my best to address. 

Thank you so very much for your time, expertise, and patience with my rambling question. I look forward to hearing how people weigh in. 

Kind Regards,

LS

The original post: /r/datahoarder by /u/EnigmatheEgg on 2024-06-11 14:02:33.

Hi all!

I have started to trust the internet less and less and have decided to save as much as I can in my own drives at home. I managed to find a Fractal design R5 with all drive sleds still there and thought it would make a great NAS Chassi. But now I'm wondering, do I need a NAS or can I just move my PC components there and just have a PC with 30+TB of space? All of the other people on the network don't have much data to store and would rather I help them anyway to save things. What other reasons would there be to have my data storage in a seperate chassi?

The original post: /r/datahoarder by /u/Yukinoooo on 2024-06-11 08:59:09.

Is it dangerous to leave 80GB on my 1TB HDD (only files without OS) ? I'm using GNU/Linux + KDE

The original post: /r/datahoarder by /u/ddd102 on 2024-06-11 06:42:01.

https://preview.redd.it/6t4uhiti1w5d1.png?width=1842&format=png&auto=webp&s=132460dc51581caaa305c1c9e1e0f859306fcf41

Hi, there.

I'm very newbie on YoYotta.

Today, I do my first copy job. 3.5 inch HDD to LTO 8 Tape by YoYotta.

But, I wonder how can I change destination folder trees. This software create just same folder trees from the source folders. I don't want that way. I want to create different folder trees on LTO Tapes which I'll do back up my data.

Anyone knows how can do that?

And is there any LTO or YoYotta user community? even though subreddit.

I need more information. Official website of YoYotta, already I checked, but I need to story from real users.

Thanks!

The original post: /r/datahoarder by /u/PrivateAd990 on 2024-06-11 04:54:27.

I'm looking for strategies, tools / software and tips to make my long journey of organizing these backups easier. Main questions at the bottom for a TLDR.

About the content:

• ~3tb total, 15 laptop backups, 12 phone backups
• there will be overlap / duplicates in content between backups
• backups contain folders I manually dragged onto the portable drive. I never plan to do a full restore of a backup.
• backups may contain photos, videos, downloads, photo editing files, code and projects I wrote, a lot of junk I'd like to scrap

Hardware:

• A recent MacBook Pro
• empty Samsung T9 4TB SSD, ~2000MB/s
• WD 2.5" passport HDD 5TB, ~110MB/s
• A second HDD with a copy of the above for redundancy

Plan:

• first bring everything on the HDD to the SSD since the SSD is way faster
• sort through everything. I need help with this part
• move the organized backup back to the HDD since SSD's aren't suitable for cold storage
• implement a plan for the 3 2 1 backup method

Questions:

1. software or tools to sort, organize, de-dupe, delete through everything on the drive. Free or paid
2. tips for how to search through everything instead of going folder by folder? I'm guessing software can help here.
3. output folder structure suggestions? Should I just flatten all backups to one? Let's say, all photos I took with my phone from all the backups to one folder? Or is that a bad idea

The original post: /r/datahoarder by /u/n4n4n4n4n on 2024-06-11 04:40:59.

(I will be using cryptomator to mask the data)

The original post: /r/datahoarder by /u/Throwaway173638o on 2024-06-11 01:39:20.

I was curious in recovering any previously-deleted and current photos off a vintage camera. The catch is that it doesn't have an SD card and that taking any additional photos after its filled starts to delete them.

It does use a 3.5 mm cord with some kind of port for vintage computers. I have no problem getting a 3.5 mm to USB cord.

Is there a similar process with data recovery for SD cards and hard drives that I can do with recovering data from the camera? Would I also need some drivers for this camera to detect too?

For context, the digital camera is a Mattel Barbie Digital Camera from 1998.

The original post: /r/datahoarder by /u/Previous_Day4842 on 2024-06-11 00:44:32.

Is one of these drives better than the other? It seems on here some people mention the WD black being total rubish. I really like these drives for the aesthetic, but have never owned one. I have owned the MyPassport models for years and years and have never had an issue. Are they equal and i'm safe to get the WD Black for aesthetics? Or would it be wiser to get the MyPassport Ultra, with the metal build and USB-C Connection?

I would be using this drive for time machine backups. Aesthetics are rather important to me, so it is a bummer that the MyPassport ultra does not come in black.

The original post: /r/datahoarder by /u/cjsalva on 2024-06-11 00:02:22.

The original post: /r/datahoarder by /u/tessereis on 2024-06-10 22:24:38.

I'm moving from various cloud drives to local backups. I really would like some suggestion from this sub.

I've 2 backups.

1. SSD 1TB - formatted to exFat. I've heard bad things about this fs but I really need this drive to be cross compatible and plug and play. This seemed to be the only sane choice.
2. HDD 2TB - formatted to ext4/zfs. Haven't decided on this yet, will probably go with zfs because it auto recovers and doesn't need to run something like fsck. Increased size for versioning.

I'm planning to keep my SSD portable (on a trip etc) and whenever I'm at home, connect it to a system (rasp pi or a regular PC) with photoprism or librephotos installed. HDD is going to be the cold storage.

I tried to follow the 3-2-1 rule but can't afford to get another drive just yet. Although, I'm thinking to get S3 Glacier for that offsite storage.

The original post: /r/datahoarder by /u/Dking2204 on 2024-06-10 21:44:27.

Coming from MacOS using Transit to Windows, I would like to move a large number of personal files from the Mac drives to the new Windows machine. What software is recommended? I keep seeing that Filezilla needs to be more secure, and I'm unsure about others. I appreciate any help you can provide.

The original post: /r/datahoarder by /u/pally_nid on 2024-06-10 21:01:22.

I barely know how to ask this question...

Would anyone know of a file copying software that can handle a source/destination like a NAS with multiple 1gb network interfaces.

Maybe its best if I describe the effect I am looking for.

Start a copy job, reach the avg-peak of 100MB/s and then the software becomes aware of this and then continues to enumerate folders as subtasks and starts these folders on the other IP addresses available from the NAS.

So, if the NAS had 2 gb/s NICs, both would be saturated at 100MB/s, rather than only one granting higher thorough-put.

Thank you for reading.

The original post: /r/datahoarder by /u/Immediate-Risk-7569 on 2024-06-10 20:15:53.

Every software I found is a trial with either a time limit or an ugly watermark.

The original post: /r/datahoarder by /u/ChumboChili on 2024-06-10 20:00:30.

Hello all -

I have a significant volume of photo dupes to work through, but I am a bit particular as to how I want to proceed through them, and so I wanted to ask about those with experience with deduplication apps.

I want to create a master set of photos, organized by year and device. Accordingly, in a serial fashion, I want to use a folder as a reference source, compare its contents to a second folder, and in the second folder I would like to be able to:

--easily see the non-duplicate photos;

--also be able to see the duplicate photos, and delete them.

I would also like to be able to identify and delete duplicates WITHIN a folder and its subfolders, although I understand that functionality is pretty standard.

One final question, is whether this can also be performed for video files.

Any recommendations to achieve this workflow would be most appreciated. Thanks all.

The original post: /r/datahoarder by /u/South-Order2046 on 2024-06-10 19:06:17.

I want to download a gallery of a creator that I have been subscribed to. I first tried download them with using "WFDownloaderApp" but it only downloaded the public images and gave me an error while trying to download the private images (subscribers only). Then, I tried my chances with gallery-dl. First, I had tried without adding any .conf file, but it gave me "API responded with 429 Too Many Requests." error. It doesn't work with any amounts of delay. Next, I added the .conf file, put my own client_id and client_secret values into that file. But it still give me the same error. How can I solve this problem? Thank you all for your answers.

The original post: /r/datahoarder by /u/th_teacher on 2024-06-10 17:13:44.

I believe nothing works well with "standard" tags, only stored in the database?

I do not need multi-user handling.

But I do want to be able to periodically export from the database and store the ratings in the FLAC tags for playlist creation

hoping other servers/players will at least use them read-only

Last played would be nice too, is played - count a thing?

The original post: /r/datahoarder by /u/4bstractals on 2024-06-10 16:49:06.

So, I just discovered PodcastBulkDownloader thanks to a recent thread, and it's got we wondering...

If I am going to start assembling a podcast hoard, what are the criteria that I might use to decide what gets included? Obviously, podcasts I like would be the primary metric -- but I can download all of those in a couple of hours, and I have a lot more space.

So... what about podcasts at risk of going behind a paywall? Podcasts of significant cultural importance? How does one best serve as a casual archivist for such a massive amount of data?

The original post: /r/datahoarder by /u/djensenteeken on 2024-06-10 16:09:33.

Hey everyone, so i have a situation i need help with.

I recently switched over from my HP Omen laptop to a new MacBook Pro. Not knowing about the data formatting situation, I backed up all my data from my old laptop onto my Seagate Expansion Portable Drive 2TB. When using my MacBook, I noticed I couldn't upload files to this harddrive because the file formatting was different.

I found on the internet that to switch the external harddrive from NTFS to APFS I can use Disk Utility to reformat the device. For this to happen however i have to backup the files that I have on my external harddrive, because to reformat it the harddrive has to be erased.

The thing is, if i back-up the files to my old HP Omen, I can't reupload the files back to my external hard drive because it has been reformatted to APFS, and the files are NTFS. I also can't back-up the files to my new MacBook, because the size of my files is way to big for the internal storage of my Macbook!

Is the only option for this really just to buy another external harddrive coded to APFS, or keep my files on my hard drive on my old dusty laptop that barely functions? This feels like a catch-22 situation to me.

Sorry if this question has a) been answered already or b) is not relevant to this subreddit. Any help is appreciated!

The original post: /r/datahoarder by /u/raafayawan on 2024-06-10 12:00:46.

Audiobooks. E-books. Comics/Manga. Magazines. Songs. Movies. Series. Podcasts. Documentaries. As a data hoarder, is there anything else to hoard?