It's A Digital Disease!

23 readers
1 users here now

This is a sub that aims at bringing data hoarders together to share their passion with like minded people.

founded 2 years ago
MODERATORS
8126
 
 
The original post: /r/datahoarder by /u/Inevitable_Airport80 on 2024-07-20 03:48:21.

Hi, fellow datahoarder, I am new here and not a native English speaker, so please bear with me.

I have recently begun collecting an ungodly amount of pono vids, and a lack of space on my NAS has forcibly stopped this addition. To cope with the ongoing addiction I have bought new HDDs and put them in my nas. I have noticed that lots of these vids are in older uncompress format and wonder if there are ways to compress them (all 20 TB of them)

So I would like to ask you fellow, which format is appropriate for good compression and quality, is there any way to process them in batches with multiple video resolutions and encoder?

I have all vids stored on my NAS and it is connected with 10gbps sfp cable to my switch, I have quite a lot of hardware available to me such as threadripper rigs, nvidia gpus and also amd gpus. I don't know where to start on this project.

Thank you in advance for the recommendation.

8127
 
 
The original post: /r/datahoarder by /u/GeraltOfRawalpindi on 2024-07-20 03:46:20.

I'm looking at this drive:

Seagate IronWolf Pro 24TB Enterprise NAS Internal HDD Hard Drive – CMR 3.5 Inch SATA 6Gb/s 7200 RPM 512MB Cache for RAID Network Attached Storage, Rescue Services (ST24000NT002)

Can anyone tell me I'd this would work as a nor al drive in my PC? I just need it to download all my steam library (or most of it) onto the drive.

Is there any implication of it being a "NAS" drive?

8128
 
 
The original post: /r/datahoarder by /u/bitpandajon on 2024-07-20 02:36:21.

Say I have (2) 12TB drives. One I want for every day use and another as a clone backup. What’s the best way to backup about once a month without having to clone the entire drive? I just want it to alter the clone to match the original, adding and removing files. I’ve been doing this manually but I can’t be sure I haven’t made a mistake along the way and something isn’t backed up.

8129
 
 
The original post: /r/datahoarder by /u/Cryogenicastronaut on 2024-07-20 01:25:24.

For one of the largest social media platforms in the world, there are surprisingly few options for people wanting to save a livestream. I would have thought there would be more demand and more people asking about this, but there are hardly any save for a few reddit threads scattered throughout the years.

There is only https://github.com/Michele0303/tiktok-live-recorder really. And this one doesn't work half the time. I've also tried using VLC to convert the .flv of the livestream to .mp4 or .ts , but it keeps stopping on its own within 7 minutes and I can't figure out why.

8130
 
 
The original post: /r/datahoarder by /u/DocJuan on 2024-07-20 01:14:56.

I am setting up a new array just for plex. My setup is an old xeon server running on proxmox. Installing truenas as VM. Plex will be installed under a ubuntu VM and point to that truenas array as storage. How reliable is the HC520? Getting the 12TB HC520 drives ( on a raidz1 pool with 8 drives with SSD cache.

Western Digital Ultrastar DC HC520 HUH721212ALE600 0F29590 12TB 7.2K RPM SATA 6Gb/s 512e 256MB 3.5" ISE Power Disable Pin Manufacturer Recertified HDD

8131
 
 
The original post: /r/datahoarder by /u/AsbestosSniffer69 on 2024-07-20 01:01:06.

Hi everyone, Sorry if this is a dumb/basic question, but I don't know a lot about SSDs/storage and could use your help.

I'm looking to buy a 1TB SSD to regularly back up my phone (Samsung Galaxy S23; UFS 4.0). My backups usually involve around 60 GB of data, but they are probably going to get larger in the future. I'm interested in a durable SSD that offers good transfer speeds and reliability for this purpose.

I'm considering the Samsung T7 Shield, SanDisk Xtreme Portable, and SanDisk Xtreme Pro Portable. The T7 Shield and Xtreme Portable offer write speeds of around 1000 MB/s, while the Xtreme Pro Portable reaches up to 2000 MB/s.

My questions are:

  • Does the difference between 1000 MB/s and 2000 MB/s make a significant practical difference for my use case? I would assume that the bottleneck would lie somewhere else.
  • Are there any other alternatives that you would recommend for a durable SSD?

I’m specifically looking for a durable SSD to have peace of mind knowing that it won’t easily break if accidentally dropped. While I understand that regular SSDs might also be sufficient, the extra durability is important for me.

I appreciate any recommendations and experiences you can share!

Thanks in advance!

8132
 
 
The original post: /r/datahoarder by /u/ShoeGod420 on 2024-07-20 00:23:20.

I have multiple folders I've downloaded from MEGA. I just got done downloading a folder and when i went to check the folder for duplicates using Awesome Duplicate Finder by dragging the folder into the program instead of the folder path showing it shows my entire Downloads folder. When i try and drag the folder into a different duplicate finding program it just doesn't work. When i tried to change the name of the folder it says the folder is no longer located in that directory even though I can clearly see it, and open it and open the files in the folder. I have multiple folders that were downloaded from MEGA where this is happening. Does anyone know what is going on? This is only happening with stuff I download from MEGA. Also I just checked and when i right click the folder and select properties it shows the folder size as 0 bytes, yet if i just hover my mouse pointer over the folder it shows as 17GB, which it should be.

8133
 
 
The original post: /r/datahoarder by /u/HG-ERIK on 2024-07-19 22:25:27.

so i have a 2 hdds and i want to use them in one smb sharing something like this:

i made a drawing :) and sorry for the noob question but i have files on the 1 hdd and i dont want to lose them with a rookie mistake

so the client only sees the server as one network location but actually in there the 2 folders are two seperate drives

https://preview.redd.it/e2t1f5kmxjdd1.png?width=1208&format=png&auto=webp&s=395c42f5d12766e5cf09825204876dae84d85fe2

8134
 
 
The original post: /r/datahoarder by /u/PapaDoomer on 2024-07-19 22:12:08.

After hard lost of data, I'm trying to get it back and save it somewhere, I'm talking about around 2tb of movies/music/images, and I wonder what should I choose, ssd, hdd or cloud?

8135
 
 
The original post: /r/datahoarder by /u/Hoagiewave on 2024-07-19 21:05:45.

I'm having a crappy time getting the dust particles off my scanner glass. I've searched a lot of different word combinations online and can only really seem to find pages that reference wiping it down with a microfiber cloth, sometimes dry, sometimes damp. I've tried this and the only thing that seems to happen is it moves the dust around and drops lint. I'm using brand new microfiber cloths.

There has to be a better solution, any ideas? I'm considering painters tape to pull it off and then wiping off any residue that might get left behind.

8136
 
 
The original post: /r/datahoarder by /u/Sapphire_01 on 2024-07-19 20:04:13.

Hi there! My family has a bunch of old photos spanning from the 90s to the early 1900s (most are between 1910-1960).

I've searched on google everywhere and a highly reccomended scanner I've seen here, the Epson FastFoto FF-680W, and similar scanners all seem like they'd bend the photos in the scanning process, which would likely cause mine to break or tear.

I can't seem to find any good scanners out there that are plainly flat and have no bends as the pictures go through them.

I am very new to this project, any and all ideas are very much appreciated! Thank you for your time

8137
 
 
The original post: /r/datahoarder by /u/Air-Flo on 2024-07-19 18:48:10.

3 copies of your data

2 storage mediums

1 offsite

Easy to understand and so far I'm doing 2-1-0 (Not including 2TB on Dropbox), and my second NAS arrived and my hard drives come next week, so once everything's copied to it and I take it to my brother's place I'll be at 3-1-1.

Just not sure how to get to 2 mediums, and seems like most the people quoting the 3-2-1 method don't actually do the 2 mediums themselves or they're willing to pay for cloud storage.

I know it means using tape or optical discs (DVDs, Blu-rays) but what are some other methods? Do SSDs count? Curious how people here are managing the 2nd medium? Maybe I could buy a shipping container of floppy disks.

8138
 
 
The original post: /r/datahoarder by /u/KabegamiAkuma on 2024-07-19 19:33:44.

Hello, i am here to ask something, how i can download all en.uesp.net include other language and view it offline?

I have try with WikiTeam with its command:

python2 dumpgenerator.py --images --xml --curonly WIKIA_LINK

But it don't download images

Yes... I have make a error with ElderScrolls Fandom, i make mistake and i am sorry.

8139
 
 
The original post: /r/datahoarder by /u/ciscam5 on 2024-07-19 16:50:52.

Hello fellow data hoarders!

I want to share a little story that made me smile.

It was three weeks ago when I received a mail from my TrueNAS machine, telling me that one of my 3 TB WD Red's (my setup is at the bottom) was dying. That server had been running 24/7 since 2012 when I started my homelab, migrating from external HDDs to a proper (Free)NAS setup with redundancy, ECC and all the next level enterprise-grade stuff. It was housed in a NZXT Source 210 in the beginning, which was already a blessing to work with, and the best I could afford at 40€.

Of course I ran Plex on the same machine and of course it was disgusting. The motherboard alone cost almost as much as my notebook, so that server better do everything I've always wanted to do at the same time all the time. I only started to use the box exclusively as a file server when I retired my notebook and put Proxmox on there. How lucky that was, because the knowledge and familiarity I gained landed me more than one job position.

Anyways, it was 2018, I was earning 4 digits and wanted to treat myself. A case that's almost 100€?! Unbelievable! But the internet said it was the best in build quality that you could get and data hoarders celebrated it as tailored to their needs. So I ordered a Fractal Design Define R5 and god, I still get goosebumps remembering the case migration.

From then on, drive replacements were not a chore anymore. I could toollessly open that beautiful heavy-duty box of metal, appreciate the sight that I experienced, slide out the drive with the correct SN, of course have the cables easily slide out with it thanks to the nice cable management options, switch and slide it back in. I would be lying, though, if I didn't admit that I'd caress and de-dust the interior every time that happened.

But back to the actual story: I replaced the defunct WD Red as usual and remembered that I've had >80% used pool capacity for some time now. I procrastinated fixing that, because I still don't have the moneys to add storage capacity in the form of 2.5" SATA SSD's or even PCIe-M.2 adapters and NVMe drives. But I do have a box of decommissioned HDDs that survived a harsh burn-in from my last job position and enough free SATA ports. Just the drive slots are all used.

In the last 2 years I helped two very good friends of mine configure and build new PCs. For both, we started looking at case options, features, manufacturers and both asked for my favorite brand. You can guess it, I said Fractal Design is neither cheap, nor light, nor fancy. But it's nice to work with, silent, has dust filters and you can actually feel that you paid an extra buck. One got a Pop (I was shocked how close their "cheap and colorful" line comes to the Define experience), the other got a Define Mini C. Both are very happy with their choice and the latter even upgraded to a North later on, the one with the wood, passing on his old Porsche of a case to his mom.

Anyways, one evening we were sitting in Teamspeak playing some Mechabellum and I lamented about Fractal Design offering replacement parts, but not anymore for my age-old Define R5. My friends said "you talk so positively about that manufacturer, why not just ask them?". I wouldn't have ever thought of that myself, don't ask me why. I went to the contact form and asked if they could help me source that part that they seemingly don't manufacture or sell anymore, as I couldn't even find it on the whole of eBay Europe.

I received a reply within 7 hours and 8 days later opened a package containing a 3x 3.5" HDD cage for my ancient Define R5. It's already doing its job but I'll want to move the cage to where the bottom fan is in my next maintenance window:

https://preview.redd.it/3ieu8q029idd1.png?width=2000&format=png&auto=webp&s=bffa435a3fc0009c4f1249ce319a6b0e05e8914a

I never expected anything to come out of this and I'm still overwhelmed that they actually read my inquiry, reached out to me and took the time to help me solve my problem. They're one of the few companies where such an exchange is still possible. And I wanted to share that story with you. Thanks for reading.

Setup
Case: Fractal Design Define R5
Mainboard: Supermicro X10SL7-F
CPU: Intel Pentium G3220
RAM: 4x Samsung 8 GB DDR3 1600 MHz ECC (on Supermicro's compatibility list)
OS: TrueNAS
ZFS pool 1: 6x 3 TB RAIDz2 (used to be only WD Red EFRX, the last used and next 3 spares are HGST Ultrastar 7K4000 SAS 3 TB)
ZFS pool 2: 2x 18 TB RAID1 (Seagate Exos 18 TB X18)
ZFS pool 3 (new): 2x 1 TB RAID1 (miscellaneous HDDs)
Also new: 1 internal 3.5" slot for a hot spare

8140
 
 
The original post: /r/datahoarder by /u/illbollocksyou on 2024-07-19 16:44:11.

Hi. Im trying to backup lotr fotr extended edition. Ive managed to backup all the other lotr discs however, neither fotr theatrical nor extended are able to backup. Especially the theatrical edition. Once i insert the theatrical edition disc into the drive, it stops reading any other discs until i boot it up 2 weeks later and try. I have no idea how thiis works but it does. Anyways. I tried backing up part 2 of fotr extended edition again and this is the error im getting

Error 'Scsi error - MEDIUM ERROR:L-EC UNCORRECTABLE ERROR' occurred while issuing SCSI command 28080..001 to device 'SPTI:\Device\CdRom0'

i get this about 10 times then

File "/MAKEMKV/AACS/DUPLICATE/CPSUnit00001.cci" was skipped

Error 'Scsi error - MEDIUM ERROR:L-EC UNCORRECTABLE ERROR' occurred while reading 'BD-RE HL-DT-ST BD-RE BH16NS40 1.05 KLLN5GE2007' at offset '655360'

Error 'Scsi error - MEDIUM ERROR:L-EC UNCORRECTABLE ERROR' occurred while reading 'BD-RE HL-DT-ST BD-RE BH16NS40 1.05 KLLN5GE2007' at offset '657408'

udf: node fe/efe failed!

for all of the internal m2ts files. Recently, this has also been happening to my inception disc which has a couple scratches on it. However, none of the lotr discs have any scratches of smears at all on the disc. I'd like to know if this is a drive problem or if this is a makemkv problem or if this is a disc problem.

edit - i retried both fotr theatrical and fotr extended part 1 makemkv says no disc inserted for both. however, when i tried other 4k discs after that it seems to work now. I also remembered another strange incident. When i tried to rip criterion's pinocchio, makemkv couldnt rip with chapters but could do it without chapters

8141
 
 
The original post: /r/datahoarder by /u/Jatts_Art on 2024-07-19 16:26:20.

I've used this software for a good few months, for Twitter/X, and never had any issues until today suddenly. In fact, just this morning it downloaded content just fine but now when I just tried to DL another user's content it keeps outputting "0 download links found" which I found to be odd... then I tried another page, and another, and another, and all of them now say "0 download links found" including pages I had downloaded properly in the past (again, like earlier today).

So I go to update the cookies and user agent information to try again and then it tells me that I needed to update it because something was outdated relating to cookies (I dont remember what it said specifically, and it has never said it again).

So THEN I go to the main page to download the newest version, which is evidently Beta-0.88 (mine is Beta-0.87) and when I activate the .exe it gives me an error saying "CreateFile failed; code 3. The system cannot find the path specified." and then it cancels the install. I re-open the program and all messages relating to updates no longer exists, despite it still being Beta-0.87, yet it still keeps saying "0 download links found"? Bruh.

With all this said, I am currently left with a broken WFDownloader that cannot be used, fixed, restored, nor updated (uninstalling & re-installing also did not work)... despite it working completely fine just a few hours ago. What's the deal here? Every few months I have to keep hunting down new methods to download and archive favorite artist content from Twitter/X and then when it breaks I have to go hunt something new down... problem is, at this point there isnt anything else so I'd really would like to try fixing this, and would appreciate any help that could be offered.

Thank you!

8142
 
 
The original post: /r/datahoarder by /u/M_a_l_t_e_s_e_r on 2024-07-19 15:55:36.

Original Title: Does anyone here have experience using sony XDCAM discs for backing up non-video files? I recently got ahold of some as well as a matching deck (conenction points are Firewire or ethernet for FTP, no usb on this model) and would like to put it to use

8143
 
 
The original post: /r/datahoarder by /u/blueman541 on 2024-07-19 13:17:21.

Due to budget costs instead of IOPS I've been using 2x6 striped mirror zfs pool to store photo/video. Upgrade two drives at a time every 1-2yrs & resilver. This approach lets me mix/match different new drive sizes over time. Been working well for over a decade.

Recently discovered used recertified/refurbished drives from ServerPartDeals/GoHardDrive for 1/3 the cost. Same price as my usual two drives upgrade I could buy six.

All drives fail new or used but unsure about AFR from SPD/GHD. 2x6 Mirrors allows 3 drive failure at best or 1 at worst. 2x6 RaidZ2 allows any 2 drive failure.

If I get a batch pair from SPD/GHD and both fail my mirrored pool is toast. If 1 dies then I am 1 drive away from total failure. Yes, I have physical media backup but restoration is a PITA.

Leaves me wondering if I should keep doing my mirrored setup or switch to RaidZ2?

8144
 
 
The original post: /r/datahoarder by /u/fleshbagMaraud3r on 2024-07-19 12:42:55.

I'm trying to backup my main HDD (G-Technology G-Drive 6TB) to two Toshiba N300. All work files: CinemaDNG original takes and original shots from events. I've read about Borg and Pika and Vorta but I'm trying to understand which software is the most reliable for what I'm trying to do and if there is some software that can test the disks after to see if they are ok or not. Thank you!

8145
 
 
The original post: /r/datahoarder by /u/Jeeeaaan on 2024-07-19 11:47:17.

Hi guys,

I need to bulk download a website. It has a login page that HTTrack doesn’t seem to get pass.

Do you know any good alternative please?

8146
 
 
The original post: /r/datahoarder by /u/yhapstixk on 2024-07-19 10:03:04.

Link in comments⬇️

8147
 
 
The original post: /r/datahoarder by /u/animeshin on 2024-07-19 08:44:22.

When archiving podcasts, I run the following yt-dlp command:

yt-dlp "URL TO RSS" -o "%(upload_date>%Y-%m-%d)s %(title)s [%(id)s].%(ext)s" --download-archive {archive.txt} --write-thumbnail --write-description --write-info-json --write-playlist-metafiles -P "FOLDER NAME"

It downloads all the episodes in the RSS, includes the description and playlist metafiles. My aim is to do a complete backup of each podcast show. Is there anything else I could add to this command do make it even more complete?

8148
 
 
The original post: /r/datahoarder by /u/monoko13 on 2024-07-19 08:24:08.

So lately Ive been trying to use it to grab from various twitters but Ive been running into an issue with the link search. Is there a way to "pause" it and then continue from where it left off? From what I can tell you can stop it but you have to reset your progress. is there a way to stop it and then come back to it later and continue searching from where you left off? Also is there a way to control just how fast it grabs things as I find that causes it to get tripped up and kicked out due to how much its grabbing all at once.

8149
 
 
The original post: /r/datahoarder by /u/Capt_Crunchy_Nut on 2024-07-19 08:18:45.

Hi all,

I'm in need of very basic storage, somewhere to store my family photos and videos. No need for cloud access or the ability to stream from this storage device whatsoever, and I'm not going to be copying heaps of data at a time so bandwidth/throughput is not a big deal (except for day one of course). It's just a content vault.

I've been looking at NAS solutions but think the most cost effective and acceptable option is a DAS plugged into my desktop. I do however want drive redundancy (RAID 1) and have read that hardware RAID on DAS devices is iffy.

Any truth to this? When this comes up people recommend software RAID which I've not researched yet.

If a DAS is OK for my purposes, what are suggested brands and models? 2 bay would be enough but I'll look for a 4 bay model just in case e.g. QNAP TR-004.

Thanks in advance.

Edit: just found the wiki entry on software RAID so will be doing that, assuming I use a DAS.

8150
 
 
The original post: /r/datahoarder by /u/rumble_you on 2024-07-19 05:43:02.
view more: ‹ prev next ›