It's A Digital Disease!

The original post: /r/datahoarder by /u/sulicadiz on 2024-06-17 19:35:31.

I just bought a Western Digital DC HC580. I don't know an effective method to know if the drive is CMR or SMR. I have read here in the forum that I should buy a CMR hard drive to use with the NAS (I have a DS923+).

Another question, before setting the NAS should I introduce all the drives I would be using?

The NAS have 4 slots, so I guess the first step would be to buy 4 hard drive then set up the unit. I suppose If I insert 2 drives then the other 2 I would lose data.

Anyway, total noob here, please help

The original post: /r/datahoarder by /u/justquestionsbud on 2024-06-17 18:43:19.

So, just found out Google Podcasts is shutting down. I loved it, because whatever podcast I wanted, I could just find it in Google Podcats, youtube-dl the whole thing, and I don't have to worry about content for months at a time. Now that it's going...seems like the end of my podcast consumption. Any ideas? Pocket Casts doesn't seem to have a browser-based option.

The original post: /r/datahoarder by /u/Bloodmoonwolf on 2024-06-17 18:21:50.

I currently have less than 800GB across 2 clouds and my laptop. I'm hitting my storage limit on Google and looking for a safe, local option. After losing everything on an old laptop that crashed, I started doing cloud storage, which is now becoming expensive.

My current laptop is an old HP and I have yet to decide between a new Windows laptop or a Chromebook. I have a Plex library I would like to expand, even bought an external DVD reader to start the library. I don't necessarily need NAS. Plugging something into the TV or my laptop would be fine when I want to watch something on Plex (which isn't very often). The same goes for when I need to do a regular backup of files. I would prefer to buy something once instead of paying a monthly subscription and to not add another constant draw on our power supply.

Most of the storage is for movies/shows, photos, and PDF scans of documents from when I went paperless. I would like to add music to this once I figure out a few things.

What type/size/brand of local storage would you recommend for my situation?

The original post: /r/datahoarder by /u/lilbud2000 on 2024-06-17 15:05:44.

Yesterday I bought my first "big" refurb hard drives (2x12TB HGST drives, upgrade from a 2TB and 4TB drive).

The current plan is to have one in my computer, and use the second as a backup with an external enclosure. Probably not the "best" way to do it, but it should suffice in the meantime.

Currently waiting for them to ship and looking into the whole testing process in the meantime.

I was wondering what would be the best way to test them, as I've read about a bunch of different ones (like smartctl, Badblocks, HD Sentinel, etc.) And it's making my head spin a bit.

I guess my questions are as followed:

1. What/how many tests need to be run on a refurb drive? I've seen some posts listing multiple long tests and others just saying a few SMART tests. Is there any general consensus?
2. Badblocks is on Linux only, would that work on something like WSL? Or would I have to get a Linux machine/VM setup? I have a Pi 3 collecting dust, could that be used?
3. How long would testing the drives take? I've seen that a full badblocks 4 pass run can take days or even a week of 24/7 running just for one drive. Does that sound right? I was thinking about using a secondary machine like my old Thinkpad if it was going to take a week. My desktop (where the drive will eventually end up) is in use daily, and I'd be a bit concerned about leaving it on but not killing the test accidentally.

I'm a bit new to all this, only having a 2TB and 4TB drive for the past few years. Any help in making sense of all this would be appreciated.

The original post: /r/datahoarder by /u/Shumhow on 2024-06-17 14:33:59.

Found a seller online selling a 1TB laptop HDD with casing for about 15 USD. Says everything is alright with the HDD, it is from Seagate and 3 years old. I have tried looking up at how old is too old for HDD but I understand there is no definite answer for the 'use'. But would it be advisable to 'buy' one which is 3 years old? I barely have any experience with this, so please do help me out! Thank you!

The original post: /r/datahoarder by /u/flac_rules on 2024-06-17 10:56:06.

I have an older Areca Raid-card. The manual claims it supports very large drives with 48 bit LBA, but i also found a google cached search result from the areca site that claims the following:

"The maximum capacity of HDDs for Areca RAID controller's old version firmware supports up to 16TB capacity. From firmware version V156-20190124...."

The newest firmware for the card is older than v156, but i don't know if this quote is for a particular card or in general, I can't find a complete changelog in the site for the changelog of the card I have.

So i know it is a bit of a long shot, but has anyone tried a larger than 16TB drive on a Areaca 1260-card or something of around that age? And did it work?

The original post: /r/datahoarder by /u/Not-The-Dark-Lord-7 on 2024-06-17 06:18:49.

Not too sure if this is the right place for this, but I want to hoard my data, so I feel like I’m on the right track. What is the best way to deal with a school-managed google account, in terms of keeping our data? Having recently graduated, I would like to download my stuff from things like Google Drive and Google Docs, as well as maybe a few emails and Google slides. What is the best way to do this? Just manually download everything? No “download all” button anywhere? Also, given that the new institution I’m going to gives students a google account, is there any way to just migrate my stuff from the old school’s account to the new one? That would be really convenient.

The original post: /r/datahoarder by /u/DeadbeatSummer13 on 2024-06-18 01:50:26.

My dataset is around 10-16tb. I plan on transferring my current externals to 1 big drive. I’m trying to decide if this working drive is going to be internal or external. Regardless, this will be backed up to a 2nd drive daily. Then, the 2nd drive will backup to backblaze daily. A private encryption key will be set on backblaze. Possibly down the road an off-site drive may be added to be backed up weekly and then disconnected and moved off-site.

Feedback is greatly appreciated. What do you think?

The original post: /r/datahoarder by /u/Few_Thought_248 on 2024-06-18 01:02:50.

The original post: /r/datahoarder by /u/cfxla on 2024-06-18 00:36:26.

hi! I'm looking for a way to use AI to summarize classes from a online course, can´t watch them all until the subscription ends, and there are too many and too long.

the videos are not on youtube, but a closed/paid platform. is there a way to use AI to do this? I know just the basics of programming. thanks!

The original post: /r/datahoarder by /u/1michaelbrown on 2024-06-17 21:08:51.

I have setup a bash script to autorip so far it is working but with errors. So how would I fix the errors or do this a better way. Errors I am having

Jun 17 20:24:15 markvm5 (udev-worker)[9984]: sr0: Process '/bin/systemctl start makemkv-rip.service' failed with exit code 1.                                                                                    
Jun 17 20:24:15 markvm5 (udev-worker)[9995]: sr1: Process '/bin/systemctl start makemkv-rip.service' failed with exit code 1.                                                                                    
Jun 17 20:26:04 markvm5 (udev-worker)[10091]: sr1: Spawned process '/bin/systemctl start makemkv-rip.service' [10094] is taking longer than 56s to complete                                                      
Jun 17 20:26:04 markvm5 systemd-udevd[467]: sr1: Worker [10091] processing SEQNUM=8075 is taking a long time                                                                                                     
Jun 17 20:28:04 markvm5 (udev-worker)[10091]: sr1: Spawned process '/bin/systemctl start makemkv-rip.service' [10094] timed out after 2min 56s, killing                                                          
Jun 17 20:28:04 markvm5 systemd-udevd[467]: sr1: Worker [10091] processing SEQNUM=8075 killed                                                                                                                    
Jun 17 20:28:04 markvm5 systemd-udevd[467]: sr1: Worker [10091] terminated by signal 9 (KILL).                                                                                                                   
Jun 17 20:41:49 markvm5 (udev-worker)[10159]: sr1: Process '/bin/systemctl start makemkv-rip.service' failed with exit code 1.                                                                                   
Jun 17 20:52:38 markvm5 (udev-worker)[10201]: sr1: Spawned process '/bin/systemctl start makemkv-rip.service' [10205] is taking longer than 47s to complete                                                      
Jun 17 20:52:38 markvm5 systemd-udevd[467]: sr1: Worker [10201] processing SEQNUM=8077 is taking a long time Jun 17 20:24:15 markvm5 (udev-worker)[9984]: sr0: Process '/bin/systemctl start makemkv-rip.service' failed with exit code 1.                                                                                    
Jun 17 20:24:15 markvm5 (udev-worker)[9995]: sr1: Process '/bin/systemctl start makemkv-rip.service' failed with exit code 1.                                                                                    
Jun 17 20:26:04 markvm5 (udev-worker)[10091]: sr1: Spawned process '/bin/systemctl start makemkv-rip.service' [10094] is taking longer than 56s to complete                                                      
Jun 17 20:26:04 markvm5 systemd-udevd[467]: sr1: Worker [10091] processing SEQNUM=8075 is taking a long time                                                                                                     
Jun 17 20:28:04 markvm5 (udev-worker)[10091]: sr1: Spawned process '/bin/systemctl start makemkv-rip.service' [10094] timed out after 2min 56s, killing                                                          
Jun 17 20:28:04 markvm5 systemd-udevd[467]: sr1: Worker [10091] processing SEQNUM=8075 killed                                                                                                                    
Jun 17 20:28:04 markvm5 systemd-udevd[467]: sr1: Worker [10091] terminated by signal 9 (KILL).                                                                                                                   
Jun 17 20:41:49 markvm5 (udev-worker)[10159]: sr1: Process '/bin/systemctl start makemkv-rip.service' failed with exit code 1.                                                                                   
Jun 17 20:52:38 markvm5 (udev-worker)[10201]: sr1: Spawned process '/bin/systemctl start makemkv-rip.service' [10205] is taking longer than 47s to complete                                                      
Jun 17 20:52:38 markvm5 systemd-udevd[467]: sr1: Worker [10201] processing SEQNUM=8077 is taking a long time 

This is the process I used I setup a udev rule

SUBSYSTEM=="block", ENV{ID_CDROM}=="1", ACTION=="change", RUN+="/bin/systemctl start makemkv-rip.service"SUBSYSTEM=="block", ENV{ID_CDROM}=="1", ACTION=="change", RUN+="/bin/systemctl start makemkv-rip.service"

and the makemkv-rip.service at

/etc/systemd/system/makemkv-rip.service`/etc/systemd/system/makemkv-rip.service

[Unit]                                                                                                                                                                                                           
Description=AutoRip CD on insertion                                                                                                                                                                              

[Service]                                                                                                                                                                                                        
Type=oneshot                                                                                                                                                                                                     
RemainAfterExit=no                                                                                                                                                                                               
ExecStart=/home/mike/autorip.sh                                                                                                                                                                                  
ExecStop=killall autorip.sh [Unit]                                                                                                                                                                                                           
Description=AutoRip CD on insertion                                                                                                                                                                              

[Service]                                                                                                                                                                                                        
Type=oneshot                                                                                                                                                                                                     
RemainAfterExit=no                                                                                                                                                                                               
ExecStart=/home/mike/autorip.sh                                                                                                                                                                                  
ExecStop=killall autorip.sh 

It's weird because the script is working but still get these errors. Also need to figure out how to trigger encoding after rip. Also in my autorip script it is finding titles in TINFO should it be finding them in CINFO.

The original post: /r/datahoarder by /u/dingwen07 on 2024-06-17 20:57:10.

The drive is two months old and has only 1336GB written.

I use it exclusively on macOS (formatted as APFS), today decided to plug into desktop and check it's S.M.A.R.T., finding it have 0E of 42.

How fucked am I? Will warranty cover this?

https://imgur.com/ZxMoOXC

The original post: /r/datahoarder by /u/pinkwonderwall on 2024-06-17 20:52:15.

Is there a way to rip captions from Patreon videos? I'm talking about Patreon videos I already have access to through a paid subscription. I like to save a video's subtitles as a text file so I can ctrl+F search for a particular word and find the moment that topic is discussed. I've tried Chrome extensions, but none of them work with Patreon. I've also looked for other posts of people asking this question, and it seems like not many people are trying to do this lol. I searched Inspect and Page Source and didn't see any obvious solutions, but I'm inexperienced with that so I may be missing something.

The original post: /r/datahoarder by /u/green__problem on 2024-06-17 17:44:24.

I have a flatbed scanner and a phone with a quality camera.

For text-heavy books I use the CamScanner app, and then scan the cover using my flatbed. Non destructive, very effective. For magazines and newspapers, the flatbed is usually enough, as the lack of a solid spine makes scanning with minimal wear very easy.

Now comes my problem: I have a lot of image-heavy books that I want to scan, but I have yet to find a good method to do so.

CamScanner is horrid at dealing with illustrations and photographs. The flatbed works alright, but not great. Because I avoid breaking the book's spine, there are always visible shadows and both text and images become a little blurry when they're close to the hinge.

I'm wondering if there's an app similar to CamScanner but more appropriate for photographs? Or a different method altogether.

I know some people melt the glue keeping the spine together, scan the pages individually, and then glue everything back on. This wouldn't work for all of the books in my collection- but I have considered trying it on a handful of them. I'm just a little scared of screwing the process up.

Thanks in advance.

The original post: /r/datahoarder by /u/Msprg on 2024-06-17 16:31:30.

Hello,

Let me preface with: I know there are a million posts about dedup tools already. Dedup by file content, checksum, attributes, similar photos, similar videos…

Yet somehow, I failed to find any tools that would be able to first filter out the majority of files that differ in filename / date / size and then on the results make sure that files are 100% surely duplicate by comparing their content.

I've tried dupeguru, alldup, freefilesync, treesize, czkawka, I tried everything! (By voidtools that is).

The point is that I'm either missing something, or that none of the tools offer the option I'm looking for.

So here I am. Once again. Seeking answer to the eternal question: How do you deal with duplicates, fellow Data Hoarders?

The original post: /r/datahoarder by /u/idle_cat on 2024-06-17 16:11:41.

On youtube, I archive livestreams of a channel. Is the live archive recording I get by using ytarchive a higher quality then the video I would get with yt-dlp that's processed afterwards? From my understanding the video goes through youtube's compression. Is the compression really strong in your opinion? I am wondering if it's worth getting the vod to save space.

Side questions:

Why do people put --format "bv*+ba/b" or something similar to get when yt-dlp already has it set to get the individual best audio and video as the default? https://github.com/yt-dlp/yt-dlp?tab=readme-ov-file#format-selection

The original post: /r/datahoarder by /u/Most_Mix_7505 on 2024-06-17 15:53:10.

What would you all do?

The original post: /r/datahoarder by /u/PuzzleHeadPistion on 2024-06-17 12:59:39.

Hi,

I'm a bit lost on how to keep all my data safe.

Currently I have an old desktop, i5-4690 + ASUS H97 Pro + 16Gb RAM, with 3Tb WD Red + 6Tb IronWolf + 8Tb Barracuda drives and 2.5GbE + 1GbE interfaces. This works as my Plex server and it's where I dump files from the desktop/laptop, it's running on Windows 10 for now, but about to switch to FreeBSD or TrueNAS (or Proxmox?) with ZFS pool.

Now I've added an Asustor AS1102TL Drivestor 2 Lite (2 bays, 1GbE) which is probably being returned for an AS1104T Drivestor 4 (4 bays, 2,5GbE). It is supposed to be a remote NAS using Wireguard, that's why I didn't care for 1GbE, but the initial backup is taking a LONG time. The price difference is only 100€ for more bays and speed (useful for full copies and full restores if needed). Here there's a 6Tb WD "white label" and an 8Tb Barracuda as single volumes (JBOD looks risky and can't use RAID with different drives).

Part of my issue is which file transfer protocol to use. NFS? My desktop and laptop are Windows, not sure NFS works properly. FTP? Or SMB? SMB is giving me speed issues, not going over 150MBps and for some reason when cloning the 6Tb IronWolf to the 6Tb WD "white label" the speed sinks to 10MBps. It's copying RAW photos and videos, like thousands of 50-100Mb files mostly.

Having file access sorted, what's your recommendation for file transfer/backup? Asustor Backup Plan? Paragon Backup and Recovery? Macrium Reflect? Or, since I already own SyncBackPro, just use that? This question is related to both, from my computers to the main NAS and main NAS to the Asustor.

A little guidance will be much appreciated, since I want to go through this once and "forget". My day job is IT PM so I know my way around a computer, but by far not an expert in this area. ty

The original post: /r/datahoarder by /u/TeamSylver on 2024-06-17 12:54:43.

Everyone here probably already knows that.

I've just had all 3 drives in my desktop PC suddenly have problems.

Thank god I can still read/write to the drives though. It's just god awful slow, especially during data transfers, where it will render the whole OS unusable until it's done.

So that was a lot of pain and agony to temporarily move everything to the spare PC and laptops I have laying around (PC has 1tb, laptops have 1.5tb and 4.5tb).

Means I now have no backups at all, since I still haven't finished setting up my work PC to be my off-site backup PC yet (it's basically manage/byo PC at my work BC it's such a small store).

Annoying as well since that PC hosts my active directory and vaultwarden as well as the file server (thankfully I had a secondary active directory server set up, but no vaultwarden).

Gotta love Crucial NVMes. All of them only 11 months old. Never again. 2 of 3 RMAs processed but I still gotta get data off of the third (OS drive) before I can post that.

Edit: Forgot to mention they are Crucial P3 Plus 4TB NVMes

The original post: /r/datahoarder by /u/mehlaterlater on 2024-06-17 07:58:36.

I have some pictures and videos from 15yrs ago. I kept moving them from usb to usb, but I heard that if I dont use it for a few years I can lose all the data. I think SSD has a similar problem because the motor can go out. So what should I use?

The original post: /r/datahoarder by /u/Arcau1 on 2024-06-17 11:53:45.

I have seen a little N100 board with a 4x NVME hat on ali express (also comes in a N305 version)

I was wondering if it was possible to maybe use the nvme > 6 sata riser cards ive seen also.

So turning this into a little 24 drive beast of a nas,

So brain trust of the community i ask you:

Is this even possible?

Would it cripple the N100?

Would the speeds on the disks be just stupid slow?

Has anyone tried something like this already and have any words of wisdom?

TIA

The original post: /r/datahoarder by /u/Void-ux on 2024-06-17 11:08:52.

Hey, I store a relatively small amount of media (movies and tv), and some of it I likely won't watch for decades.

Most of it is 1080p, and I keep my fav latest TV shows in 4k. Is there any way to losslessly compress this media? From what I've heard 1080p is best in h.264, which it is. The 4k stuff could be converted to h.265 10-bit, and I have done this with HandBrake, but I'm skeptical of how lossless it is since the file size reductions are ridiculous efficient.

The original post: /r/datahoarder by /u/the_Athereon on 2024-06-17 10:56:41.

Genuine question. How many of you have had a year as bad as mine so far?

5 failures. 1 DOA

Parity 1 and 2 went in January

The first Replacement Drive was DOA

Data Disks 5, 8 and 11 have since failed.

I've been able to recover 90% of the data through the use of my backups and catching the problem in time. But seriously. 6 drives have died on me this year. And we're only half way through the year.

They're dying so frequently that I can barely afford to replace them.

Now. For the details.

Parity Disk 2 had a physical fault of some kind. Reallocated sector counts when from 0 to 256 in one night.

Parity 1 had a controller board failure (This will be a common cause. I've figured out the problem since this happened.)

Data Disk 5 kicked the bucket spectacularly. The Seek Error Rate went from 85% accurate, which is the average in my server due to how many disks are in there. To 1% in the span of 3 days. Making it infuriatingly slow to get any data off the drive but still possible.

Data 8 and 11 both experienced controller board failures. Strange drop outs in connection, hang ups, read and write error flags despite no data corruption either reading or writing. Obviously I couldn't trust those drives anymore.

But this thing is, only 2 of these failures are genuine faults. The other 3 are my fault.

The drives that had controller board failures, at least some of them, were due to how much pressure was being put on the sata connectors when I closed the side panel. Yes, I'm serious. In any other circumstance, the Define R6 would have ample room for sata power and data cables at the rear of the case. But when you have 11 drives and all their cables back there, the thickness of the noise dampening foam presses into those cables and puts dangerous amounts of pressure on the connectors.

I proved this by running read checks on the "failing drives" with and without the side panel on. With it off, 1 drive had errors 100% of the time. With it on, all drives showed the same errors. Errors which disappeared when I removed the side panel... SMH.

So now I need to replace yet more drives, the cables and the case.

My server is a bottomless money pit. It has to be.

The original post: /r/datahoarder by /u/Foreign_Factor4011 on 2024-06-17 09:26:57.

I know this might not be the right community to ask this question, so if the moderators need to delete this post, go ahead.

I have a lot of music on my hard drive (we're talking 1000+ songs) and I'd like to organize everything into playlists.

Each .mp3 file has metadata and I have software to organize playlists. I think I'll create the folder like this:

Music/Genre/Artist/Album/.mp3 Files

Do you think there's a better way to organize it? I did some math and there would be at least 20% artists with maybe 1 music. There's another problem: some tracks aren't even part of an album. How can I improve this, if possible? Is there a better way to do it?

The original post: /r/datahoarder by /u/Yukinoooo on 2024-06-17 08:26:46.

I want my HDD to be efficient, good performance, fast, no error messages like impossible to read folders or read mode, bad sectors... If I want to use my HDD, it's for media files like photos, videos, music...