It's A Digital Disease!

The original post: /r/datahoarder by /u/CommercialDue1397 on 2024-05-30 04:10:15.

Hi,

I have a budget of around $5,000 CAD and would like to try to back up my data as well as I can. I would like to have 2 backups, 1 onsite 1 offsite.

I have around 35 TB of data that I need to back up, and another 32 TB that I'd like to dedicate to new data.

I'm looking into making a NAS build with RAID1 and creating the same build twice (once offsite, once where I live) but I really don't think that's cost-effective. In terms of drives - eyeing WD Red at the moment.

I'm pretty sure this is a bad idea. I just don't know what would be considered a good idea. I don't know much about this sort of stuff and I'm tired of having 20 hard drives that could fail at any time. I've been lucky that my drives haven't failed in 10 years+ (besides one) but I don't want to take that risk anymore.

Budget could increase theoreticaly to $8,000 CAD if needed but I'm trying to stay under $5,000 as much as I can.

Please help suggest any solutions that I should look into. I don't have a good local network where I live right now, but I am going to move soon and will be able to get 10gbit inside the new place + 1gbit symmetrical offsite.

Thank you!

The original post: /r/datahoarder by /u/ligerzeronz on 2024-05-30 03:33:47.

So currently, I have 4 portable hard drives which i am not using. These have family photos/videos/documents.

I am wanting to use these drives in the next 2 months but has to be completely empty.

I am also now at a phase where i need stuff backed up just in case (3TB on one NAS, same data as above)

I am not worried about it being cold storage as I have now uploaded these to Youtube and Google Photos. What would you guys recommend? A few I've heard is Backblaze and crashplan but i am completely new in this realm

The original post: /r/datahoarder by /u/capitalsix on 2024-05-30 03:32:48.

I am a small packrat, and have about 36TB of unsorted data spanning about 20 spinners and SSDs.

I would like to index and sort the directory structures of each into a consolidated list.

I've looked at DupGuru and Everything as well as few others in the DH Wiki but there still seems to be some camps of just code it yourself.

I'm just short of writing a Powershell script to index each with, Drive Serial, Drive Capacity, Drive Model, file name, file size, file hash, and log each file on each drive into a consolidated list and reading that raw data into Excel. Sorting things and stacking things instead of making piles is a nightmare for me, and a skill I'm slowly grinding through learning. What software would you personally use to scan and categorize your (unsorted) files across several hard drives that aren't connected simultaneously?

The original post: /r/datahoarder by /u/r4nchy on 2024-05-30 02:16:29.

Recently came across this llama-fs tool. They say it can automatically organize messed up files in proper directory structure, it can rename files etc. This is done by the AI which will be reading the files.

What do you guys think about it ?

Will it be useful, or will it create its own mess ?

The original post: /r/datahoarder by /u/pea_gravel on 2024-05-30 01:19:11.

There's this old GitHub repository that I'd like to have its Wiki and Discussions backed up. Do you guys know if it's possible to have a copy of it in markdown or any other format?

Thanks

The original post: /r/datahoarder by /u/Ispeakforthelorax on 2024-05-29 22:36:20.

Just discovered this sub, and wanted recommendations on external hard drives that will last!

For a bit of background, my laptop is currently filled with all my stuff (personal documents, stuff from my undergrad, photos, videos, phone back ups, etc.). I currently just have a back up on a WD 1 TB HDD (I know I'm a bad boy for not following the 3-2-1 method, but I am trying to now!). I've been using this HDD for the past 8 years (and it saved my ass when my laptop died on me twice), and I am recently starting to hear sounds from it when it's running and it got me worried to find a replacement.

All my data is ~250 GB, and I am looking for recommendations on what are some good external hard drives I can use! Although I am considering online cloud services, but I'm not sure if I want to pay a yearly subscription which has roughly the same annual cost similar to an external hard drive. I would prefer to pay for 2 hard drives and have it last for couple of years instead, rinse and repeat.

I am looking to buy 2 hard drives, but am having a hard time decide which ones. I don't care about transfer speeds, and am looking for 500GB to 1 TB memory. The thing I care most about is that it lasts. I know there isn't any guarantee a that a external drive will last and could die any moment, but I would prefer it to last at least 5 years (hopefully 10 years) before I find a replacement. I generally do a back up 3-4 times a year, and let it collect dust on my shelf for the rest of the year.

I was thinking of buying a HDD since I thought those last longer than SSDs, but apparently a bunch of websites on Google is telling me otherwise? That today's SSDs now last as long as HDDs or longer? Has SSDs surpassed HDDs in the past decade?

Right now, I got my eyes on:

Samsung T7 1 TB Portable SSD

LaCie Rugged Mini 1 TB (SSD)

Seagate STHN1000400 1TB Backup Plus Slim Portable Drive

WD - Easystore 1TB External USB 3.0 Portable Drive

I'm thinking I may do 1 HDD and 1 SSD cause why not lmao. I would be open to any suggestions on how to go about this!

I would love to hear your thoughts and recommendations from your personal experiences!

The original post: /r/datahoarder by /u/Wise_Leather_9159 on 2024-05-29 21:37:24.

Okay I've been banging my head on every wall and surface of the Internet trying to solve a problem, I want to extract some 3D models from a mobile game so I can do some renders with em in SFM but everything I've tried just seems to come up empty or just simply not work. Does anyone have any advice, how to's etc etc???

The original post: /r/datahoarder by /u/Gabba_Rama on 2024-05-29 20:43:52.

Taking place online, may be of interest. https://us02web.zoom.us/meeting/register/tZEscuihpjwvGdT4RvNn7xPQbc0KsnpLHCGT#/registration

The original post: /r/datahoarder by /u/Ambitious-Republic-8 on 2024-05-29 19:02:29.

I'm looking for advice on setting up a reliable backup system for my web server, which hosts 200TB of data. Currently, I have a second dedicated server as a backup. This means that if my primary server is hacked or damaged, the backup server can immediately take over and bring the site back online.

Now, I'm considering a third backup solution using HDD SATA drives. My idea is to back up my website onto hard drives that will be stored in a secure location at a hosting company. If my primary server or its drives fail, the hosting staff can simply insert the backup drives into a new server to restore my website quickly. Additionally, I plan to keep another copy of my website on HDD drives at my home.

This approach seems cost-effective and could potentially minimize downtime by avoiding the need to transfer large amounts of data from the cloud.

I'd appreciate any insights or recommendations on setting up this type of backup system. Are there specific considerations or best practices I should be aware of?

Thank you for your assistance.

The original post: /r/datahoarder by /u/rnourse on 2024-05-29 17:17:58.

Looking for some ideas from the hive mind here. The department I support is retiring a number of applications and wants to archive the data for regulatory and compliance reasons. But these apps are a mixture of structured and unstructured data and since at least one of them is a SaaS app we wont have the ability to simply leave the old system running in RO mode. I'm trying to develop a shortlist of commercial products that can handle both the db tables+schemas along with pdf files, emails and documents in a single tool, ideally via a single pane of glass.

Has anyone here had a similar challenge and if so, what types of tools did you consider? Cloud as a target is fine.

The original post: /r/datahoarder by /u/ColdDijon on 2024-05-29 16:33:25.

I am non-techy and I would like to download all the images from a specific website for personal archiving.

Is there a way I could know exactly just how much storage/size a website has before I download the whole website/images, and is there a way to just download the images, and have them downloaded automatically categorized like it does on the site? The site is a photo gallery.

Thanks to whoever can help!

The original post: /r/datahoarder by /u/Idostuffgud on 2024-05-29 16:13:42.

Currently running into an issue with adding another drive to my hoard. Any time I connect a new drive I get a BSOD Internal power error. I remove the new HDD and the error goes away. I have tried other driver other power connectors but nothing beside removing the newly connected HDD fixes it.

Currently running:

Os: Windows Server 2022 Standard v 21H2

HDD count 10

SSd count 2

PSU 850watt

Any idea what I am doing wrong ?

The original post: /r/datahoarder by /u/sezayesh on 2024-05-29 15:26:58.

Hello guys. Noob here.

I just bought a seagate exos 10TB and i think i made a mistake. I read someone saying about iron wolf drives that you should reduce start/stop events on these drives because they are build for 24/7 operation. Does it mean they are not suitable for pc use?(Since you keep turning them on and off) And is it the same with exos dar drives?

Thank you

The original post: /r/datahoarder by /u/Individual-Many-5784 on 2024-05-29 08:11:12.

Just started properly getting into the rabbit hole that is datahoarding; for the past couple years I have only been storing my relevant files on external storage devices and using RapidCRC to generate CRC32 hashes for the files within each directory (such that I can pick up on any file corruption issues during quarterly checks of my data). While I have since decided to build myself a proper NAS to serve as my primary storage (with external storage and a cloud service serving as the backups), a couple questions still remain in regards to my current practices with external storage:

To my understanding, there are far better hashing algorithms than CRC32 but would they provide any tangible benefits over CRC32 solely from a data corruption perspective (e.g. a lower chance of hash remaining the same in the event of corruption, even if unlikely to begin with)?

(I'm probably overthinking this one, but) does it matter if I have one checksum file per directory (that contains all the file hashes in said directory) as opposed to an individual file per item?

Lastly, would there be a more efficient method of checking the directories (opening checksum files to verify file integrity) than doing so manually? I don't have much practical knowledge with running scripts and the like, but am willing to learn if necessary.

Thanks for reading and appreciate the help! :)

The original post: /r/datahoarder by /u/YouWide5985 on 2024-05-29 07:43:29.

I've spent the past few weeks doing this, and only now do I realize that I have the data hoarding disease.

The original post: /r/datahoarder by /u/SuperiorSkills on 2024-05-29 07:29:08.

I used Easycap for my camcorder to digitalize them, but the audio isn't in sync at all

i spent so much time and money already :(

The original post: /r/datahoarder by /u/Ill-Alternative6308 on 2024-05-29 05:49:05.

I have a total of 30tb. 3tb(2tb wd external, 1tb 2.5hdd) on a nas with another 3tb(1tb 3.5, 2tb 3.5) to back it up. 3 tb on my laptop(1tb nvme ssd, 2tb 2.5hdd), 5tb on my xbox(1tb internal, 2tb nvme with enclosure, 2tb external), 2tb( on my pc, with this i have about 11tb worth of hard drives(ranging from 320gb to 4tb) and 3tb worth of micro sd cards and usb drives lying around(from 2gb to 250gb). How can i use this extra storage?

The original post: /r/datahoarder by /u/LauraAmerica on 2024-05-29 21:46:55.

I recently went back to my parents' (overseas) and found my backup (WD 3TB My Book). I got all hyped, it enclosures the treasures from my golden years. Sadly, it doesn't seem to work properly anymore —even when it was stored in a dry fresh dark place (for over a decade).

When I plugged it in the computer it was able to read it without issue. Knowing the fragility of the situation, I started to copy the whole drive instead of browsing it. After 10% or so it stopped working. I re-plugged it in a different port and retried with similar results (with a progress equally low). After that it became hard to read it so I stopped —I didn't want to damage it by forcing a huge transfer in an unstable condition.

My current plan is to try to access it from some Linux distro and, if I can read it, copy by folders (everything is organized in main categories).

I don't think my old friend has much time left for trial and errors, so I'd like to check with you. I'm sure you'll have better ideas than my current barbaric plan.

Thank you, this is important for me, and I highly appreciate and advise.

The original post: /r/datahoarder by /u/Archivist_Goals on 2024-05-29 20:22:44.

I wanted to create a separate post that further elaborates on what was mentioned in this post's comments from others. And, I will tag the original post's author u/KeptinGL6:

Just to clarify for everyone, MPF is the overall GUI.

There are 3 main utilities from different people that are used in the Media Preservation Frontend program:

1. Redumper
2. AARU
3. Disc Image Creator (DIC)

~~DIC is actively being phased out. Redumper is pretty much the gold standard right now. AARU is also very well developed. DIC's original developer sort-of abandoned it and it's regarded as a legacy dumper utility and will be phased out at some point.~~

Edit: Some clarification. The original developer behind DIC didn't abandon it. Redumper is being chosen over DIC where applicable and/or possible.

These programs' inception originate to the Redump.org, Video Game Preservation Collective, Gaming Alexandria and Hit Save! organizations and communities where, through Discord, a ton of collaboration has taken hold for proactive media preservation, the world over. It's a very sizable, but niche community of dedicated people very much akin to the VHS Decode and Domesday Laser Disc duplicator projects.

Links

https://www.preservegames.org/

https://www.gamingalexandria.com/wp/

https://hitsave.org/

Discords

https://discord.com/invite/AHTfxQV

https://discord.com/invite/dQhd6d7

https://discord.com/invite/TCKT6uA

The Video Game History Foundation is also an invaluable resource

https://gamehistory.org/

These utilities are not solely aimed at preserving video/PC games. They can be used to preserve all forms of media. EAC, Exact Audio Copy, is a bit dated because it can not grab, for example, compact disc subcode

https://en.wikipedia.org/wiki/Compact_Disc_subcode

Redumper can be found here:

https://github.com/superg/redumper

BOS (Binary Object Scanner) is an integral part of MPF for detection of copy protection mechanisms and can be found here: https://github.com/SabreTools/BinaryObjectScanner

While EAC is a decent utility, it lacks the ability to grab all possible information written to a an optical disc. Redumper/AARU have the ability to grab even more.

The original post: /r/datahoarder by /u/desperado491 on 2024-05-29 17:00:31.

My setup is:

Playing VHS tapes on a Sony VHS/DVD combo with component jacks running into a Sony DCR-TRV 340 Camcorder and then via firewire into my Windows 7, capturing into WinDV.

The biggest issue I've noticed is dropped frames- 33 frames dropped over the course of a 2 hour tape, another tape dropped 14 frames. This causes it to go out of sync. Is this the result of my computer? The tapes? I'm not sure how these dropped frames can be avoided and any help would be greatly appreciated.

Another issue are these moving lines going up and down the screen. I've attached a video below that shows it. Adjusting the tracking doesn't seem to solve the issue. I first saw this issue in OBS but it's still happening in WinDV. Can anyone help identify these lines so I know what's causing them?

Thanks so much!!

https://youtu.be/xtHPgzeOorc

The original post: /r/datahoarder by /u/jplank1983 on 2024-05-29 13:41:04.

I'm interested in downloading all shows from a particular musician from the Live Music Archive on archive.org in FLAC format. Is there an efficient way to do this? The way I've been doing this so far is to navigate to each show's page and download the flac files manually. But, this is tedious as there are many, many shows. I'm wondering if there's some tool that would allow me to do the same thing more easily. I am on a Windows 11 machine, in case that's relevant.

The original post: /r/datahoarder by /u/Ihavefinancialissues on 2024-05-29 12:38:45.

Storage option for on-site backup ? This is part of a backup plan with off site and cloud backup.

View Poll

The original post: /r/datahoarder by /u/shadowoflight on 2024-05-29 11:36:08.

Hello,

Totally new to NAS.

I typically just add drives to my desktop, so in that sense I'm already super used to treating each drive individually.

Been considering an NAS because sometimes the random drive spin-up causing the whole system to slow down or pause. Can be annoying, esp when gaming, that's why I'm considering shifting all my data storage off-pc.

I thought it was just a matter of getting a NAS and plugging the drives in, but it seems like in a NAS, it is considered a single storage drive regardless of the number of drives it has?

I'm assuming you can split up the storage, but that's still not using the drives individually yes?

Or is there a way/setting you can set so that you don't have to use any kind of RAID and just have the system treat each drive as, well, a drive?

The original post: /r/datahoarder by /u/Amareiuzin on 2024-05-29 11:26:54.

I need to do some moves among my drives, and I'm afraid one of them might die/fail during the juggling of files, where I could potentially lose up to 1TB of personal data (basically my life story).

So as a safety net, I was thinking send everything to OneDrive and then after all is said and done, delete from the cloud.

Problem is I don't trust their deletion, or them with my data for a few days even, so I was thinking maybe I could zip everything with a password, but I heard that's easily crackable, so I thought I'd ask any data hoarder here if you happen to ever use the cloud: how do you keep your stuff safe in the cloud? what encryption do you use?

The original post: /r/datahoarder by /u/Direct-Arm-5041 on 2024-05-29 07:28:50.

My external hard drive is formatted exFAT and has tremendous trouble transferring files that are over 15 gigs in size.

Would formatting it to NTFS solve this problem?

The hard drive basically just flops and shuts down if I try to transfer a file that is 20 gigs or larger.

Whereas my NTFS SSD can handle 100 gig files with no problem. Granted it's an SSD so it's faster, but is the formatting a factor here?