It's A Digital Disease!

The original post: /r/datahoarder by /u/DollopOfCrazy on 2024-07-02 00:16:46.

Hey crew! I want to un-sync some of the decade worth of photos in my phone to free up storage, but my ten year old MacBook has fewer gigs than my new iPhone. My question:

(1) is it possible to buy an external HD and back up my iTunes to the external (the laptop itself doesn’t have near enough storage so I cannot first back it up to the laptop and then move it to the external).

Or is it more worthwhile to (2) buy a new laptop and just do it the normal way. This is mor expensive by about $900 but maybe just easier and more sustainable (I’ve read some older posts on hesitation to use externals here).

The original post: /r/datahoarder by /u/dekekun on 2024-07-01 23:48:14.

I've had a linux based NAS/fileserver running on PC hardware for around 15 years now, upgrading it along the way, and have always during that time been of the view that SATA cards are untrustworthy, and wherever possible onboard SATA should be preferred.

I'm at the point where I want to start scrounging for parts for the next upgrade and have found that almost all motherboards except the very top of the line models no longer have enough SATA ports to feed my drive addiction.

So curious what the prevailing opinion is these days - are PCIe sata cards reliable enough these days, or should I suck it up and buy a z790 board to get enough onboard SATA?

The original post: /r/datahoarder by /u/globogym1 on 2024-07-01 23:29:17.

Hey all, longtime listener, first time caller.

I inherited a collection of about 30,000 35mm slides documenting some very important local history.

Over the past 5 or so years I’ve gotten scans of most, if not all of them using my Nikon Super Coolscan 4000 with Nikon SF-210 attachment and VueScan.

Recently I came into possession of another 200 or so slides that fill holes in the original collection of 30,000 slides. I just upgraded to Windows 11, and when I pull up VueScan it no longer detects my scanner. Windows doesn’t see it either.

I’ve downloaded the most recent drivers for my FireWire card and device manager says the PCIe card is working properly.

Nikon Support told me they no longer support that scanner and therefore no longer have the software available for download.

Does anyone here have any advice? I’ve also reached out to Ed Hamerick with VueScan. But I was hoping to hit this from multiple angles to see what works.

Thank you all, I love this community. I’m hopeful someone else can help!

The original post: /r/datahoarder by /u/Maratocarde on 2024-07-01 21:35:46.

The original post: /r/datahoarder by /u/123ranchdressing on 2024-07-01 19:46:14.

Hi all, I want to preface by saying that I am NOT A CODER and this is basically my only experience using command prompts, so please ELI5. I’ve tried a couple different things and none of them have worked.

Fandom images are all hosted on static.wikia.nocookie.net/[wiki name here]/ and I was able to find them on Wayback Machine under this domain, all being listed as image files. I put this into the Wayback Machine Downloader, however, all of these images downloaded as index.html files with a bunch of random code/text and no images. No clue why.

So then, I was able to find a list of image links by using [http://web.archive.org/cdx/search/cdx?url=url name here]*&output=txt. I tried to insert this list into JDownloader to download current versions, and for a smaller wiki, it worked like a charm, but for a larger one (roughly 100k links), nothing worked. I couldn’t copy and paste the links into the pop up box for links, I couldn’t drag the links, I couldn’t drag a txt file with the links— it would freeze for a second, then go on like nothing happened.

Where do I go from here? Any advice would be helpful. Thank you!

The original post: /r/datahoarder by /u/GermanSchanzeler on 2024-07-01 21:02:10.

I just leave this here, assuming you might be interested in advancing storage technologies:

https://www.youtube.com/watch?v=zKY1EzNqLDY

sidenote: needed to delete and re-post, because I first wrote 200 Gigabyte. Still perplexed, didn't ecpect such a progress in research of passive offline data storage.

The original post: /r/datahoarder by /u/TheOneFromEarth on 2024-07-01 20:22:02.

I want to save the Twitter accounts of two people who are very important to me. Are there any working tools for automatically getting every tweet, image and video? The only Tools I found were outdated and not working.

The original post: /r/datahoarder by /u/cjsalva on 2024-07-01 20:17:18.

The original post: /r/datahoarder by /u/eyal8r on 2024-07-01 19:46:25.

I need to transfer ~3TB of data to my cloud service, and my ISP caps my data usage so it'd cost me quite a bit if I did it from home. I need a location that offers free wifi so I can hookup my laptop/backup drive and transfer it to the cloud. Any suggestions? Public Library? McDonalds? Starbucks? A Hotel lobby?

The original post: /r/datahoarder by /u/CrazyBax on 2024-07-01 19:22:24.

I just bought my first large external hard drive that I plan to use for video storage for when I edit with my MacBook.

I’ve read (within the past 20 mins) there’s a big controversy whether to leave it running or not.

I plan to use it 2-3 times a week so I assume leaving it running is ideal? Again not really sure.

However when I unplugged the usb from my MacBook, I think it turns off due to lack of noise, even though the external is still plugged into an outlet as well.

So my question is, when disconnecting the external hard drive from my MacBook (usb connection) does it turn off? Or is it still powered in since it’s connected to an outlet?

Im pretty mobile with my MacBook and I bring it all over the place. Meaning I won’t be able to keep the usb plugged in at all times.

The original post: /r/datahoarder by /u/mactep66 on 2024-07-01 18:01:46.

It keeps giving me trouble in real loads and passing tests and idk why.

The original post: /r/datahoarder by /u/lostmsu on 2024-07-01 17:34:02.

I have a program that generates quite a lot of data (a few TB/year), and I am certain I will need to keep this data forever. I already have virtual disks in 2 locations built with ReFS and 3 or 4 column storage spaces. What I am looking for is a reliable tool that would periodically sweep the program's output directory, take the files from there to my existing disks on both machines, periodically check integrity, and warn me of any issues whatsoever. Is there any tool that can do that?

The original post: /r/datahoarder by /u/Dougolicious on 2024-07-01 17:08:38.

I'm reading about bit rot here and realizing that I've been dealing with it for decades.

Is there any utility which can scan disks (or files, or folder structures) for errors? I assume file and block checksums.

Is there any utility which can repair these given multiple copies of the same files - or large folder structures? As I have my main archives copied (as folder structures) on 2-3 different disks.

Same thing for old Cd/DVDs I've burned, it would be nice to mark those that have errors.

The original post: /r/datahoarder by /u/SciFiIsMyFirstLove on 2024-07-01 16:31:47.

It all started with my Steam and Gogs Games collection about 4 years ago and at that point I had two 110Tb Raid 6 Arrays on Supermicro dual CPU gear using 4TB SAS3 HGST drives.

Then about twelve months after that I started to get real sick and I was told that I would need both lungs replaced within twelve months and the likely hood was that I wouldn't survive the surgery, so it took a couple of weeks but I had my friends clear out all my sever gear and drive racks and I spent a lot of time from that point forward doing nothing for about two years... waiting to die.

I finally got to see a specialist and after talking to me he completely changed the medications I was on and while I still suffer from stage four C.O.P.D my ability to breathe went from 21% lung function to 43% lung function, it turned out the the medications I had been put on were fighting each other and their effects were polar opposites.

So I gave away $60,000 U.S.D worth of gear because of idiot doctors.

So now I have started again today I brought my brand spanking new 15 disk NAS, it features an AMD 7700X on an ASRock Steel Legend X670E with a PCIe bifurcation card to allow a 9361-8i and an HP 24 connection HP SAS3 expander to run on the PCIe 5x16 slot.

It has 64GB DDR5-6000 ram @ 30-36-36-76 timings, the RAID controller also has a Cachevault and Battery Back Up unit or BBU.

Although the X670E based board has a 2.5GB and 1GB Ethernet Connections I have a 10Gb Card for it so I will be adding that to the PCIE x 4 slot.

On this I will be running a 130TB array configured using 15 10TB SATA3 disks and will have an additional two spare for swaps.

My use case , I am presently downloading the complete continental United States Satellite MAPS as 1m Digital Elevation Models and 1/9th ARC second models ( as the 1Ms are incomplete )

I then intend to create a complete map of height elevation data by laying down the 1/9th arc second data and then overlaying the 1M data where available to get the most accurate map possible.

This I can then cut into various chunks of height maps for any game that I see fit to do so.

Beyond that I am actually looking at the most efficient way to store mapping data for when I create my own game since with a bit of luck I'm now not going to drop dead soon.

I am still waiting for one question to be answered though and that is if I can fit on the 9361-8I a connection to connect it to a:

U.2 Drive.

U.3 Drive

NVMe

If anyone knows please let me know.

The original post: /r/datahoarder by /u/Lexaraj on 2024-07-01 15:22:17.

I'm slowly working on my hoarding and have a a 12tb external HDD as well as two extra 4tb internal HDDs. I'm looking at getting higher capacity drives to start archiving more data.

I don't have the space/setup to currently run a NAS array, so I'd just be putting the drive in my PC for the time being. Is this okay to do with drives specifically labeled as NAS drives or will I run into issues?

Any other advice is welcome!

The original post: /r/datahoarder by /u/Jatm4 on 2024-07-01 14:47:02.

Hi there, to solve my storage problems, I'm planning to buy a 4-bay docking station (no RAID) to start setting up a homemade DAS station with new Toshiba N300 drives (18TB). But I found different options on Amazon that look the same but have very different prices (even though they're the same brand). For example, there's this docking station (https://a.co/d/067FRrlm) for $99.99, 4 bays and separate switches, and then there's this one (https://a.co/d/06xl1fUB), same brand, also 4 bays but $229.99 (plus $58 for import charges). Why such a huge price difference? I missing something?

The original post: /r/datahoarder by /u/mikhailuchan on 2024-07-01 14:35:47.

Is it possible to retrieve a list of all youtube channels?

The original post: /r/datahoarder by /u/Zealousideal-Spring3 on 2024-07-01 14:35:35.

I recently bought 2 new 8TB HHDs to replace the existing drives in my media server. I am struggling to pick a filesystem. My plan is to use one drive for data, and the other drive as a backup using Kopia (the OS is located on a SSD, the data drive will only hold media). I have a read a lot of good (and some bad) things about BTRFS, but much of it seems to relate to RAID setups, which I will not have. Are there any real advantages of using BTRFS in my case over something more basic, like EXT4? I am newer to using Ubuntu and don't really have experience with RAID setups.

The original post: /r/datahoarder by /u/Independent-Swim-838 on 2024-07-01 14:12:08.

Hi everyone,

I know this question has been asked a lot of time but please bear with me.

I am working for a small organisation that creates video content. The contents are stored in a local on-premise server with a replication factor of 2. However, my plan is to keep an offsite backup. The total content is around 30TB and will increase with time at the rate of 3-4TB/ year.

My questions :

1. What kind of backup solution should I look for? I am looking at Backblaze B2 for now.

2. I don't have to retrieve the data. The only time I have to retrieve is if the onpremise server goes down.

The original post: /r/datahoarder by /u/NecessaryEvil-BMC on 2024-07-01 14:01:02.

The company I work for has several locations that routinely takes pictures of items being built. This is the standard, and has been mostly issue free. I ran into a location in South Carolina that had taken nearly 1.5 terabyte's worth of pictures, and were running low on the 2TB drive of that server.

https://sourceforge.net/projects/icompress/ was able to compress things down to a couple hundred gigabytes. I now run that tool monthly on systems, and have it target anything larger than 2MB. Works great.

Unfortunately, the Chicago location doesn't do what everyone else is doing. That's an issue for management to fix, which hasn't happened. In the mean time I'm stuck with them using nearly 3TB out of the 4TB they've been alotted because they're walking around taking video instead of pictures of whatever's important.

While I'd definitely prefer to just have them get an external drive, move the files, and ignore it, we're expected to be taking and maintaining backups of things.

Is there a tool that can do what the Mass Image Compressor is doing? I give it a folder, and it goes through and compresses the AVIs? I know I won't get near the return that I do for pictures, but there are thousands of videos that I'm having to deal with. I'm not looking to maintain 4k video or something...the videos are mostly a walkaround of a vehicle, and focusing on some placard that gives details like serial numbers and stuff. All stuff that would be better suited to pictures, but that's a separate issue.

The original post: /r/datahoarder by /u/idontgiveafuckthough on 2024-07-01 13:17:20.

I know this has been asked hundreds of times here but still I didn't have luck with any of the suggested options. Most of the stuff I have saved is media (vedios/pics) and there's really a lot of them so bulk downloading everything at once would be the best option here. I have tried downloading all the links fetched via Reddit Manager with Jdownloader but it just ignores like 70 percent of the links, because it doesn't have a built-in reddit plugin or whatever. I have also tried a few command line programs from GitHub and as someone who's not into programming, it's literally impossible to understand what is needed to be done. I mean it. There are multiple other python scripts on GitHub that have proper guides and only a couple of minutes is required to understand how to work with them. But the how-to-use guides of those reddit downloader ones are so fucking lazy...

Please let me know if there are any working options with GUI programs or at least a command line program that comes with a fucking usage instructions lol.

The original post: /r/datahoarder by /u/AstronautEmpty9060 on 2024-07-01 12:32:24.

I found some cheap as shit drives, and they are data centre drives that have been cleaned and all data removed. Anyway, am I in for a world of hurt getting some of these drives?

The only thing I'll be storing is tv/movies/books etc. So most of it can be redownloaded and not that big a deal if I lose them.

It'd save me like $700 if I got those drives. $90 for a 12TB drive versus 300$ from a shop.

What would you do? Has anyone bought some of these drives off amazon, and did they last a while?

The original post: /r/datahoarder by /u/swaggersouls694 on 2024-07-01 02:37:52.

Hey guys, I'm looking to diy my old pc into a NAS for live video editing.

It's a 3700x, gtx 1660, basic b450m mobo, ddr4 ram, supports m.2 nvme and has a couple of sata ports (I'll also attach an m.2 expansion card and a 10gbe network card that goes to a 10G switch)

Purpose would be for 2 editors to edit live off the NAS (Working with 4-8 cams shot in 4K H.264, and the occasional 6k RED raw)

I'm thinking all active projects live on the ssds (raid 0 with daily backups) and everything else on hdds with redundancy and offsite backups where needed

Ultimately, I require speed, stability and less headaches to deal with after initial setup. I would greatly appreciate any recommendations.

The original post: /r/datahoarder by /u/bravemenrun on 2024-07-01 11:23:01.

The original post: /r/datahoarder by /u/Bern_Down_the_DNC on 2024-07-01 10:22:36.

All the photos on the iphone are separated into randomly named folders. I would like to be able to use some sort of software to organize the photos into a list by date and then drag all photos in certain date range off of the iphone onto the PC at once.

How would you all recommend doing this?

Thank you.