The original post: /r/datahoarder by /u/assult78 on 2025-02-27 15:05:58.

Idk if this is exactly news but I just wanted to share this. 2 years back I saw temu had ssd for $70 and if I got a certain number of new people to join the app I could get it for free and I did that. It was a 1tb drive and at first I thought it was just falsely written but rn I have 800 gb worth of games stored on it. And it plays games off of it perfectly fine. And it’s going 2 years strong with zero data loss and even the games I haven’t touched once since they went on the ssd are perfectly fine and able to launch without any issues. I just thought this was pretty cool and wanted to share.

The original post: /r/datahoarder by /u/the-i on 2025-02-27 14:51:03.

Hello,

Is there a way to create a small hash "snapshot" (md5sums file essentially) of a directory (and all its files, subdirectories, etc) and then compare that to the directory and show any changed files, missing files, new files that aren't in the original hash list, etc.?

Ideally a Windows GUI program.

Plenty of tools exist to create hash files of directories and files (md5sums for example), and plenty of tools will then verify the files in that hash file, but I can't find a tool that will compare a hash file to a directory and show the differences (i.e. newly added files that are not in the hash but are on the disk, missing files that are in the hash but no longer on the disk, etc)

Basically what I want is a tool like FreeFileSync except instead of comparing two directories, you can compare a directory to a md5sums file (or some kind of similar hash list/"snapshot")

I want to be able to run the tool on a directory to create a "snapshot" then then run it again later and quickly see that several new files have been added, or that one or more files have been removed, or that the contents of some files have changed, etc. Pretty much exactly what tools like FreeFileSync do, except replacing one side of the comparison with some kind of hash file/snapshot.

The "snapshot" needs to be small (like a hash list, md5sums file etc) not a parity file or complete data-containing snapshot or copy of the directory or anything large like that. I just want a quick, small and simple way to figure out what (if anything) has changed in a directory, not actually protect and recover the data.

The original post: /r/datahoarder by /u/MericaFTWs on 2025-02-27 14:01:07.

I have 4, 8tb Seagate SMR drives that I use for backup. I recently put them in my safe for better 3,2,1 redundancy. I used to have them next to my PC. However, my safe is bolted near a window unit (quite large AC). My question is, with the safe bolted to the same stud and wall as the AC unit will the vibrations from the unit harm the drives that I have in there. I was reading online that the heads aren't over the platter when not in use. So using this information, it should be fine, right?

Should I keep them where they are or move them? Thanks for the advice

The original post: /r/datahoarder by /u/qubedView on 2025-02-27 13:38:33.

Torrent of: https://lil.law.harvard.edu/blog/2025/02/06/announcing-data-gov-archive/

Size: 16.7TB

Pieces: 1068540 (16.0 MiB)

Magnet: magnet:?xt=urn:btih:723b73855e90447f02a6dfa70fa4343cfc6c5fb0&dn=data.gov&tr=udp%3a%2f%2ftracker.openbittorrent.com%3a80%2fannounce&tr=udp%3a%2f%2ftracker.opentrackr.org%3a1337%2fannounce&tr=udp%3a%2f%2ftracker.coppersurfer.tk%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.leechers-paradise.org%3a6969%2fannounce

Torrent contains the tarred contents of Harvard's S3 bucket containing their data.gov files.

Please forgive me, this is the first time I've made a torrent, and it's a doozy. Feedback very welcome!

Why tar files? This contains 300k+ directories of data, with a lot of very long file names. My first attempt at the torrent resulted in a 1.4GB file. Even tarred, I had to run mktorrent -l 24 to get a chunk count that wouldn't be rejected by clients.

The original post: /r/datahoarder by /u/nbtm_sh on 2025-02-27 13:00:46.

With all of the new legislation being passed in the USA, I fear that sites like e621 may be forced to purge content.

I feel that it's important to back these sites up, not just for the NSFW artwork but because a lot of SFW content is hosted there too, and often is in the highest quality possible.

If it isn't being archived, I can build and run a script on my server. e621.net have been very generous and allow JSON formatted searches and post results without any sort of API key. They advertise having ~8tb of content. I have enough free space to store all of this.

The original post: /r/datahoarder by /u/nemuro87 on 2025-02-27 11:19:44.

If my information is correct, SSD, USB sticks, SD cards and less so spinning rust, are susceptible to data loss if left unpowered over a few months or 1-2 years.

I have some external SSD enclosures, USB drives of various ages and some 2.5in HDD as well as 3.5in HDDs.

From what I read, to re-power or refresh a usb drive so you can safely power it off and forget about it for the next 6-24 months, plugging it in once in a while isn't enough to prevent data loss, and instead you should do a complete READ (not write) surface test.

If you're on windows there are plenty of free options like https://victoria.en.lo4d.com/windows

But how do you do this if you're on a mac?

I could only find a paid solution, DriveDx, but from what I gathered, it can't be used on external drives, since it can only surface test the internal drive.

If free is not an option, I'm open to paid options as long as it's a one time fee and not subscriptions.

The original post: /r/datahoarder by /u/Dex_Ultima on 2025-02-27 10:40:52.

I wanted to ask you hoarders if there's a nice and simple way to save pics, stories and videos from instagram.

I've been using this exstention for years: https://chromewebstore.google.com/detail/turbo-downloader-for-inst/cpgaheeihidjmolbakklolchdplenjai , It still work as of today 27/02/2025, but it feels unmaintained and unreliable.

The great feature about it, tough, is that it downloads files (using singers as an example of a username) with the template rihanna_1234567_8901444 (i think it's a timestamp) which makes it easier to organize the various pics and videos. Not only that, but it can also bulk download entire profiles, and it automatically creates a folder named rihanna inside a directory of your choice.

For example, you chose the directory Desktop/instagram.

Inside that directory, if you bulk downalod the profiles of rihanna , drake and bonjovi , it will create three folders with the usernames, and place all the stuff there. It basically auto-organizes itself.

So you'll have, for example

Desktop/instagram/drake

drake_1234567_8901444

drake_1256346_4534534

.....

Desktop/instagram/rihanna

rihanna_1234567_8901444

rihanna_1256346_4534534

.....

In your experience, is there a software, preferrably open source and maintained, that has the capability I described? Tried various softwares from Github and many don't work/ can't do what Turbo Downloader does so easily.

Thank you in advance for you responses.

The original post: /r/datahoarder by /u/FlowerfawnCreations on 2025-02-27 07:24:11.

Hello! I'm hoping this is the right place to ask about this sort of thing, because I am at a loss right now hahaha

I have three elusive German Watership Down games, and I wanted to try and archive them so other people from the fandom could play them without having to look for them online and paying a bunch of money. I'm not very familiar with archiving CDs or anything, but after a few tutorials, I got the first CD done with little to no problems and had my friend try it out to see if it would work (which it did).

But now I've been having issues with getting the ISOs from the other two, On ImgBurn, both the 2nd and 3rd CDs couldn't be turned into an ISO, only a .bin file. When I try to run that it stops at "track 3" and never moves forward after that. I tried a couple of other things afterward and none of those worked, but after examining the files in the CDs, I noticed that both the 2nd and 3rd CDs have a folder called "internet", which the 1st one doesn't have. On the inside of the folders both of them have a file called internet.exe (which my computer is registering as a virus), along with a readme file that says something about internet safety in German. Point is, I think it's those files (or at least the internet.exe files) that are making it so I can't archive the two. I don't know how I can get rid of them though because I don't have the right permissions to delete them, so has anyone had any experience similar to this or knows how I can get around it so I can archive my two other CDs? I will be super grateful for any help!

files for the first cd

files for the second cd

files for the third cd

The original post: /r/datahoarder by /u/Derpy1984 on 2025-02-27 07:07:43.

I f**********cked up and need help. I have 7 WD external drives ranging from 8-22tb that I shucked to store more efficiently and a couple docks to utilize them when needed. I'm trying to consolidate and organize my data and these drives are not only running at literally 1/200th of the speed they used to but they make my computer virtually useless when in operation. I attempted to move 4tb earlier today. The read/write on these usually hovers around 200mbps when in their enclosures and now they're between 70-105mbps. What's worse is the transfer got down to 1.85g and slowed to .5-1mbps. I still have the enclosures that I can try and put them all back into. Is it worth doing that or are there better docks or hubs I can get? Right now I'm using both Insignia and Sabrent docks and a cenmate 3 port hub for a Plex server (which seems to be working okay).

Any help would be appreciated. I can't work like this!

The original post: /r/datahoarder by /u/isademigod on 2025-02-27 06:13:39.

I pulled like ~50 512gb m.2 drives out of computers that were heading to recycling. I'd really like to put them into one big flash array for seeding Linux isos, because the random read on my main NAS raidZ array is abysmal and a major bottleneck.

I don't really care about the raw read performance, because I only care about random read and anything would be better than the 20mb/s my NAS caps out at currently.

Is there like, a 24x USB 3.0 m.2 reader? A pcie card that can do splits on splits on splits? Whatever dirtiest way you can think of to connect a literal bucket of SSDs, I'd like to hear it.

Btw, they're all small form factor m.2 (2230) if that gives me more options.

The original post: /r/datahoarder by /u/maniksar on 2025-02-27 06:02:30.

Background

I have 5x Seagate IronWolf drives that are 10TB each. I have been using them in my NAS for a few years now.

The power on hours on 4 of them are ~58k and the last one is ~15k

I want to upgrade to larger drives and I need help deciding what to do with the current ones.

Option 1: Sell

I don’t think they’re gonna fetch me any significant amount of money but I’d like to sell them to someone who has use for it.

If I were to go down this route, what would be a fair price per drive?

Option 2: Give away

I routinely give away slightly old homelab equipment to members of the community who are getting started and wouldn’t mind giving these drives away if they’re not worth selling.

Option 3: eWaste

If they are so bad that no one would want them even for free, I’ll just go ahead and drop them at a nearby eWaste center.

As for options 1 and 2, I have a lot of packaging material from server part deals that I’m confident I can safely ship it anywhere within the US.

I’d appreciate the community’s thoughts on my options.

The original post: /r/datahoarder by /u/Such-Bench-3199 on 2025-02-27 04:23:22.

From my previous posts, I have finally, finally!!!! finished cloning my 16TB WD Home Duo onto a Seagate stock standard, no fans, non-networked external drive (I have a Mac, and I use Carbon Copy Cloner, so to the best of my ability I am 90% certain everything has been transferred)

I am forced to never use the drive ever again, as it continues to "phone home" anytime it overheats and gets "low on storage" anytime it is 5TB and lower, (previous owner is my dad and he gives me crap every time and sends me the emails.

I am reluctant to do what I have to do, but soon I will be turning it off for the last time, removing the drives, and connecting them one-by-one (there are only two) reformatting them, and using them individually to offload storage from my other drives, mainly focusing on a ton of documentaries I have accumulated over the years.

The original post: /r/datahoarder by /u/No_Distribution_7368 on 2025-02-27 04:07:42.

I currently have a desktop with four of the large old school style hard drives in it and I use drivepool to merge them into one big hard drive virtually. I would like to switch to a mini pc which obviously means I'll need some sort of external enclosure for these drives. The 4 or 5 bay usb enclosures I've seen mostly have mediocre to poor reviews. I've heard stability is an issue with usb attached storage. What's the best stable way to do this? I don't need extreme speeds, I just want it to be as fast as the hard drives are natively through sata I guess, which is drive limited.

The original post: /r/datahoarder by /u/Beginning-Rock2675 on 2025-02-27 02:07:36.

I recently purchased this PSU based on a recommendation from a Reddit discussion regarding its suitability for a multi HDD configuration. I would appreciate it if someone with expertise could evaluate the specifications and inform me whether this PSU poses a risk of damaging my external HDDs or if it is adequately compatible for my setup.

https://a.co/d/gpSNjI5

The original post: /r/datahoarder by /u/Ratpussy_ on 2025-02-27 01:50:17.

I'm using Teracopy to check file transfers in archiving projects. It's picking up files as 'skipped' that are definitely there when I check them manually. Weirdly, the email folder (emails filed to project using mail manager add-on) are duplicating.

My colleague has reached out to support, just wondering if anyone else has experience these gremlins in the data before?

The original post: /r/datahoarder by /u/guitar673 on 2025-02-26 23:47:23.

I have an old Mac mini 2018 that I no longer need and am planning on repurposing as a remote offsite backup system for myself. I was hoping I could get some critiques on the robustness and security of my plan.

My source computer is a Mac mini M4. The destination computer, the Mac mini 2018, will live at a relative’s house hardlined into their router. Both computers will run Tailscale VPN with tailnet lock enabled. The destination computer will have an 8TB hard drive plugged into it to use for storage of the backups. The hard drive will be encrypted through Disk Utility.

My main purpose for the remote backup is to prevent against theft and fire/disaster at my home. The source computer already has a local Time Machine backup connected to it which takes care of my versioning and general backup needs. I don’t think I need any versioning or snapshots on the destination computer since it will just be a failsafe for my data in case of total loss at my home. I’ll use ChronoSync on the source computer to sync my main drive plus two external drives to the destination computer.

As far as security goes, I have a few questions:

-Since ChronoSync doesn’t encrypt the data on the drive, is it necessary to keep the 8TB hard drive ejected when I’m not doing a backup?

-Should I have a firewall on both computers?

-Should I have FileVault turned on?

The original post: /r/datahoarder by /u/puzzleandwonder on 2025-02-27 01:15:14.

Hey all. TL;DR at the bottom of the post.

So last year I built a NAS for a Plex server. It's in a Fractal Design Node 804 and runs on an MSI MAG Mortar WIFI II B760M motherboard. I overbuilt it beyond Plex needs with the future idea/goal in mind of adding a GPU later for/as an upgrade from iPad to PC for golf simulator software. That still will be the case, but the primary purpose for the addition of the GPU now is for an AI rig. I bought an RTX 3090Ti and this thing is big, it's occupying well more space than I anticipated when I bought the Node 804 case to begin with.

But that's not the biggest problem (at least not yet, we'll see how well the case and fans handle the heat). There are 3 PCIe4 x4 m.2 slots on the mobo, one that apparently runs its lanes straight through the CPU, and the other two that I guess route through the chipset first. Of the 3, only the CPU-direct one is not blocked by this massive GPU now. Before adding the GPU I had a 512gb SSD in the CPU-lane slot that was the primary (and only) cache drive for the unRAID OS, and one of the other two was an m.2-to-SATA expansion card to accommodate 3 more hard drives after the 4 SATA slots that are directly on the board were filled up (I have a total of 7 HDD's currently in this box). Not only are the other 2 m.2 slots covered by the GPU and no longer make an m.2-to-SATA expansion card possible (there's not even enough space if I were to do right angle SATA cords coming off the expansion card), but the GPU is also blocking access to the only other PCI-E slot (which only is x4 anyway, whereas the GPU slot is PCIe 5.0 x16). My understanding (correct me if I'm wrong) is that if I were to get an HBA/SAS card in that other PCIe slot then it would have the same bandwidth capability as the x4 m.2 slots, correct? But, with TWO of the m.2 slots no longer usable for a SATA expansion card, and the PCIe x4 slot also blocked, that only leaves the primary CPU-lane slot as even an option for the SATA expansion card.

TL;DR. If I were to use the m.2 slot with x4 lanes connected directly to the CPU (the primary slot) for the SATA expansion card so that there is space above the GPU for the SATA cables to plug in, and then used both of the other m.2 slots for the old 512gb cache drive as well as the new Samsung 990 Pro 1tb ssd I bought to upgrade the primary cache drive with, will I legitimately be shooting myself in the foot in terms of latency/speed of access by putting the SATA expansion card in that primary spot? The HDDs contain the massive Plex library and my photo archive. I bought 64gb of upgraded DDR5 RAM and this new SSD to try and minimize bottlenecks with the AI/LLM use, but I'm unsure of what the real world performance difference will be if I bought this super fast SSD and RAM only to then plug it in to lanes that route through the chipset first. If it DOES make a big difference, then I'm looking at having to buy a new motherboard AND case in order to not give up any hard drives and maximize SSD cache access speed.

Thoughts??? thanks :)

The original post: /r/datahoarder by /u/agoosetime on 2025-02-26 22:55:14.

Building a new PC, looking for new drives. Got three M.2 slots, 2 x gen 4, 1 x gen 5

Already got a Crucial T500 1TB for the OS as there should be no need for persistent write speed.

Looking for a 4TB gen 5 for games and whatever the best price per GB is for plex for the remaining gen 4.

Thank you for suggestions.

The original post: /r/datahoarder by /u/RepressedTraas on 2025-02-26 22:16:29.

Hi all, I'm looking for a storage app that basically meets the following criteria:

-Large storage space (500GB+)

-Easily accessible for editing from multiple ends-Fail

-proof / includes mirrored backup

Any feedback is appreciated! Thanks!

The original post: /r/datahoarder by /u/Ryan1188 on 2025-02-26 19:46:33.

Any objections to a 10 x 28TB RAIDZ2 using the "recertified" 28TB Seagate HAMR drives? (ST28000NM000C) 90% read 10% write, data stored files will be very large, sequential read/write data.

Alternatives? I already have two of these as general storage disks in my desktop but I just built my first TrueNAS scale system that has 12 bays (Silverstone RM61-312) and I need to fill the bays!

I have a Asustor lockerstor10 gen 3 with 10 x 24tb Seagate exos drives I purchased at retail and they are in a mdadm btrfs raid 10 which is unnecessary for my use case. It's all I felt comfortable using given my options with Asustor's ADM OS. I plan on reconfiguring the lockerstor as my backup unit for the things I wouldn't want to lose.

I thought about getting 24tb drives to compliment the drives I already have so I can mash everything together in my truenas system and leave a few for the Asustor backup unit. But, the recertified 24tb drives are all quite old now and I have no idea how much use (if any) they have on them or what was wrong with them. I feel like the 28TB recertified drives were either unboxed/returns from OEMS or manufacturer culls of some sort?

Anyone with more knowledge on this? Thanks!

The original post: /r/datahoarder by /u/hmmqzaz on 2025-02-26 19:07:48.

Gonna do a backup. So I have a NAS drive with two raid 1s, with my most important files on it. I’ll just do a snapshot and verify checksums while I’m doing it. And forget the cloud; I’ll leave the last copy in my cousin’s safe.

I want to put it in three places. It’s only like 4tb. What makes the most sense? I thought I’d keep working off the NAS, keep one portable HDD in my safe, and leave one portable HDD with my cousin. Does that make sense?

Looking at WD my passport ultra for the backups. Any other suggestions? Maybe one of them should be ruggedized, like a Lacie?

The original post: /r/datahoarder by /u/thesentrygamer on 2025-02-26 18:28:05.

Hey all! Just sharing my recent experience with the fallen king of hard disks: Western Digital. Due to frequent commutes between home and campus, I felt it necessary to bring my data with me on an external device. I went with WD for a 5TB USB3 device because I had very good experiences with them in the past. This drive never really ended up serving this purpose and within a month was hooked up to my main system as an additional drive instead, but within a year of purchase, the drive started to spit out CRC Check errors and I began to lose data integrity.

I immediately contacted Support to get an RMA set up, where the CS Rep I received became insistent that I not only was unaware of how hard drives worked, but that I for some reason had a mac, sending me picture diagrams on how to check on whether or not the device was plugged in and visible in Disk Utility. I never once mentioned Macintosh, nor did I make any sort of claim that the drive wasn't showing up, yet it took half an hour for this rep to finally concede that my drive was faulty and begin the process of RMA. upon which he wasted no time trying to extract more money from me to repair an $150 defective product by pushing me on an "Advanced RMA" where I pay extra to get a hold placed on my card and they send me a drive first before I send it back.

It took another 15 minutes to convince this rep that I did not want to pay them extra to honor my warranty agreement, and he eventually led me down to the Standard RMA process. From here, it took a normal turn for exactly 1 week as your typical box, ship, and wait RMA experience occurs, except I waited a month and a half for my replacement drive to be shipped in the first place! I called them regularly to ask where the hell it was, and I kept getting the same unhelpful response: "We've put a mark on your incident and will have it out the door ASAP"

tl;dr WD dropped the RMA ball so hard that I refuse to buy from them again

EDIT: Wow, after getting this drive back, I've noticed that the default formatting is in exfat with an allocation unit size of 1MB, on a drive marketed towards storing games. I only noticed this as I began moving my files back onto it and discovered that the drive was using a Terrabyte more in space than I had given in files. Are they actively trying to make their drives hold less in reality?

EDIT 2: exfat is not inherently a problem, as I understand the benefits for a general audience, but the default allocation for exfat drives this size is 128KB, they made a conscious choice to change that to 1MB

The original post: /r/datahoarder by /u/artherthe3rd on 2025-02-26 17:14:58.

Hi all!

I'm a filmmaker and I'm attempting to grapple with the production side of an upcoming film.

Basically, over the course of a few months we will be generating an estimated 64TB of video that we will need to be able to safely store, backup reasonably well, and travel with. Additionally, this is a very tight budget production, so I'm trying to tackle this is the most cost conscious way possible.

While it would be nice, the data doesn't need to be particularly quick to access and can even be partially offline. We would just need access to the most recent 24hrs for cataloging purposes.

To keep costs and complexity down, at the moment I'm considering simply utilizing a 2x bay HDD dock (like a StarTech station) paired with 8x 16TB drives (like the WD Red Pros). Each drive would be formatted individually in sequence, and when not actively being transferred to would be stored in a pelican case with foam cutouts. The backup drives would be written to at basically the same time as the primary drive (So straight off the recording media) but would be stored in a separate pelican case. These cases would then be flown back to the office.

The obvious problem with this is simply that the footage will be incredibly frustrating to access, however once back in the office I imagine I could use something like a Dell R730XD to load up all of the disks simultaneously. While offloading the footage, I also intend to create a set of proxies stored to an external SSD (Likely a T5 evo) so we can catalog footage a bit quicker and go back to review things.

While this solution is about as low-tech as it can get, is there anything inherently wrong about it I'm stupidly overlooking? I would love to be able to setup a large NAS on the ship and be able to have uploads happening from multiple machines and edit off of it, but I don't think this would be feasible both pricing wise and space wise.

Last question, if not utilizing a NAS the drive obviously can't be "brand agnostic" and will need to be NTFS or MacOS Extended Journaled. While I know that Paragon provides software for either OS to open either format, I can't imagine this is fully ideal. At the moment we don't know what OS will be utilized in a final edit.

TL;DR: What's the cheapest safe and compact way to store 64TB of footage that will slowly be generated over the course of a month or two?

The original post: /r/datahoarder by /u/PricePerGig on 2025-02-26 17:09:55.

The original post: /r/datahoarder by /u/Afraid-Raspberry-3 on 2025-02-26 16:59:18.

So I got a Qnap TS-673A and want to populate it with drives to get minim 24 TB usable space. Since the NAs will be in my homeoffice I do not want it to be so loud that it becomes distracting, since besides working I sometimes chill, read and game in this room. From working in datacenters in the past I know that enterprise drives can be really noisy.

Now it is really hard do find good data on reliability and noise. It seems the general consesus is that something like a Toshiba 12TB N300 is "quite" but when I look at the data they claim up to 36db when r/w which is above what some of WDs offers claim 29-30db.

In the past some smaller capicity drives seemed to be quiter so I was thinking more smaller drives but I doubt that the noise reduction is that good when using 6 vs 3 drives if the 6 drives indivually are a bit quiter. Can any of you fine people share some of your experience? It would be much appreciated.