It's A Digital Disease!

The original post: /r/datahoarder by /u/El_Chupachichis on 2024-06-18 23:08:22.

I keep contemplating getting more storage, possibly a NAS. But I'm not doing actual streaming, just collecting an ever larger amount of images, RAW and jpg (I'm an event photographer hobbyist).

I would look at the NAS online and see perhaps a cheap 4 bay NAS, then look at the reviews and see a lot of complaints. Seems like getting a reasonably reliable NAS would be more like getting a high end desktop.

For those digital hoarders who don't have a lot of streamable data, do you prefer NAS or just a big desktop with a lot of drive slots, and maybe a software RAID? I tend to be a cheapskate so historically it's always been "buy another drive that's larger and copy stuff over" but I really need to start thinking long term.

The original post: /r/datahoarder by /u/CorvusTheCryptid on 2024-06-18 22:58:05.

The issue is as I describe in the title. I've never had a problem like this before! There's a single file on the device when I plug it in, titled "USBC ¬÷", with no file extension. It's a huge file so I assume that fixing it will allow me access to my files, which have somehow merged into this singular, huge file. Please help, I can't afford to lose these pictures!

The original post: /r/datahoarder by /u/Pommes254 on 2024-06-18 22:12:05.

I am running ytdlp to archive a large amount of channels and i am kinda worried to contaminate my archive data with add injected versions.

The thing is so far i havent seen a single one with it in about 500 videos, i just finished downloading via proxies in germany, uk and japan (downloaded videos that are about 1-2 weeks old that i already have and compared length via script)

How many users really get videos served with direct injected adds currently?

And in what regions?

I hope a feature gets merged into ytdlp that checks video lengths and alerts if it detects more than +/- 1 sec compared to a known DB like sponsorblock.

The original post: /r/datahoarder by /u/dashcash853 on 2024-06-18 22:04:07.

Is the dpi a huge deal when it comes to this, I know some do 1200 dpi but a lot of the ones in my price range are 600 dpi.

The original post: /r/datahoarder by /u/Run_the_Line on 2024-06-18 21:23:16.

The original post: /r/datahoarder by /u/laktakk on 2024-06-18 21:05:21.

The original post: /r/datahoarder by /u/kkgmgfn on 2024-06-18 20:02:41.

Trigger Warning : This post is for DataHoarders + SFF enthusiasts. So please don't come saying get a 2TB NVME, get a 2.5 SATA SSD and yap yap yap.. And I have other systems too. One EATX, One MFF, Two SFF and One HTPC. So I am not a new builder. Posting here since it will be more appropriate than posting in Homelab or DataHoarder sub.

As we know it we have very limited SFF cases with HDD support. Manufacturers hardly make them anymore.

Some options that we have today:

Node 304: Outdated and front panel is very restrictive. 92mm fans in front are noisy as they are 3pin. In my country I can't find any 92mm fan.

Jonsbo N series: Pat on the back for Jonsbo on launching several SFF NAS cases back to back. They feel like they have restrictive airflow. They aren't available in my country.

SAMA IM 01 and its copies: Supports 4 - 5. A cheap knock of this is available. But is an option.

So my question is have you guys though about squeezing extra HDDs in cases like Deepcool CH 160, Coolermaster NR200P etc. For example if I can use rear fan mount for 120 AIO intake then does 120 fan slot above have motherboard has hinges for 3. 5"HDD? Does it have space to tuck one below a dual slot dual fan GPU. I know 1 3.5" HDD can go on front panel when I use a SFX PSU. Similarly for NR200P. More suggestions are welcome. I'll use a Noctua L9i or 120 AIO as I have then lying around.

I wish this post to be open for coming years so people can get ideas and inspiration from the comments.

The original post: /r/datahoarder by /u/Foris4 on 2024-06-18 20:01:58.

Hi. I want to ask for help.

I have a hard time finding a case that has 8-bay while having enough space for my graphic card (minimum 32cm, 4 slots thick) and cpu cooler (Noctua NH-D15), ATX motherboard.

I can use pci-e riser if there is a way to secure GC. I'm not interested in cutting a case for extra space (eg. cutting off a 5.25" bay), I don't have the skills nor tools for it.

The original post: /r/datahoarder by /u/tranrep on 2024-06-18 19:17:56.

Hey all, trying to get myself situated and not the most tech-savvy person so apologies in advance if I'm missing the mark. For context, I'm mostly a hobbyist photographer that wants to just keep my data safe and I don't believe I'd have a need for most of what a NAS offers, so I'm looking into DAS/JBOD as a solution. I currently have around 6TB of photo/video I consider "important" enough to backup.

My current setup is just a single 14 TB WD External HD which is not really being backed up anywhere so I'd like to improve my setup.

At the core, I'm currently planning on doing the following:

• Buying a new 8TB HDD that I'll be "working off of", and moving all of my existing data into it
• Buying a 4 bay DAS, either using 2 drives for now for RAID1 or buying 4 for RAID4/5 (??) to periodically mirror data from the single 8TB drive onto.
• Using Backblaze to backup my PC + DAS.

Does this setup make sense for my needs? If it does make sense, does anybody have any particular product recommendations for the DAS/JBOD and if there's any specific thing to look for in the type of drive(s) to purchase? Is this perhaps overkill for the use case? Please let me know if I'm not providing enough information, thanks.

The original post: /r/datahoarder by /u/WaspPaperInc on 2024-06-18 17:15:07.

I have a Nokia C5-00 phone which have a transfer data feature via bluetooth but only for "compatible" Nokia devices, how can i solve this issue (via bluetooth mocking or something like that)

The original post: /r/datahoarder by /u/nlj1978 on 2024-06-18 17:09:54.

I have setup a Jellyfin server and have been successfully ripping CDs to flac using EAC.

EAC does have one issue I haven't figured out. There doesn't seem to be a way to have the contents of one CD ripped into its own folder in the destination.

Jellyfin's file structure wants each CD in its own folder.

Is there a method to accomplish this I just haven't found yet?

If not is there another ripping software that can do this?

The original post: /r/datahoarder by /u/xavierhollis on 2024-06-18 17:07:31.

Often I find myself checking out a gallery or a post on reddit that has multiple images. By clicking on the thumbnails I can open up larger versions of the same images. But if I want to save the images it gets tedious and time consuming having to go through them all one by one, opening up dozens of webpages or scrolling through each image to save them.

Is there an app that will basically bulk download ALL the images from a gallery, not simply the thumbnails on that one page, but the higher res images the thumbnails link to?

The original post: /r/datahoarder by /u/Head-Ordinary-4349 on 2024-06-18 16:40:43.

Does anyone here have any experience translating your local Hydrus network onto a publicly available server? I have made a local database of images which I would like to share and have editable by anyone publicly with a link, however I am very inexperienced in this sort of thing. The user resources describe a hacky server component that can serve your database over https (as also mentioned here), but honestly I have no idea where to even start. I'm wondering if anyone would be able to point me in the right direction or possibly even give me a bit of guidance on how I should proceed? Thanks!

The original post: /r/datahoarder by /u/ECrispy on 2024-06-18 15:13:30.

This is for home desktop usage, not servers/data centers where XFS is far more common.

performance - in every test I could find, XFS is near the top, beating btrfs/ext4. Its esp good for parallel workloads and almost everything on a modern desktop is like that. The only perf concern I read about is it used to have higher cpu usage for updating metadata but I believe thats been fixed and no longer relevant?

(I think for most users, performance in benchmarks may not be noticeable and other features matter more, but its still an important consideration)

SSD/OS installs - XFS is almost as fast as f2fs for these. I see no reason why anyone would use f2fs on anything other than a sd card or on any NAND device with wear leveling.

CoW/snapshots - this is no doubt a very powerful feature of zfs/btrfs. But I see very little mention of reflinks/snapshots on XFS which can achieve a lot of this. They are not atomic but enough to satisfy a lot of use cases. I don't see support for this in the usual tools like snapper/timeshift either. XFS also has support for deduping. All of this comes without the usual cost of CoW

other features - dynamic inodes (on ext4 an inode for every 16kb/256kb is wasteful, even if most people never notice it), automatic fsck, journalling (sure, copied from ext3, but thats not a bad thing)

stability/reliability - I don't think there should be any doubt about this. Its a proven enterprise class fs with a hallowed pedigree and reputation, is now backed by RHEL and has probably seen more active development than most other file systems.

The biggest factor seems to be that the default ext4 is good enough, and frankly most people will not care or know about, and should not care, about the underlying fs. There are also distros like Fedora/OpenSuse that used to use XFS as the default and have switched to btrfs. I don't know of anything that uses XFS as default except unRaid now - unRaid is used to manage TBs by home users and that probably says something.

The only concerns I've found are -

a) it doesn't support shrinking a volume. how common is this anyway? I've never seen any home user need to do this, 99% of the time you only need this when you are installing another OS on the same ssd/hdd and need to shrink your current /, which is an advanced use case.

b)supposedly XFS doesn't handle hw failures. Even on this I found no consensus - some people say its risky and can corrupt with no recovery, others say even with a forced shutdown its safe. I'm not sure if its any less robust than ext4/btrfs? Is this actually a concern these days?

The original post: /r/datahoarder by /u/wiadrovit on 2024-06-18 14:55:31.

Hey there Guys,

I've bought a recertified EXOS X18 12TB for my NAS. The drive isn't exactly loud and it performs as I would expect, but there's one thing that I can't walk past - it acts as if it was doing something, even when completely idle.

The drive lives in my Sabrent DS-SC5B hdd enclosure which I've been using for over a year now and which I'm very satisfied with.

Video: https://imgur.com/a/o39cfd8 (it's the second one from the top - as you can see its LED is blinking in a regular manner).

I've read that EXOS drives have their own APM feature which can be disabled using seachest tool. I've managed to successfully disable both EPC as well as the power balance feature, but that didn't change anything, the LED still blinks and I can hear a regular, gentle cracking sound as if something was read/written.

The drive is mounted on my Debian installation and I've confirmed that no process is using it. What's weird is that activities stop as soon as I unmount the drive (and start again as soon as I mount it back).

In fact, this is a second drive that does the same thing. I've returned previous one to the seller as I've felt something is not right with it. Both were manufactured (or rather reassembled?) back in March 2024.

I should add that I have another EXOS drive (X16 16TB sitting in my backup device) and it doesn't act this way.

Is it normal for these drives or do you think I should return this one as well and go for something else?

It isn't that much annoying, I am just worried that if the head keeps flying all the time, the drive will wear sooner and might die prematurely.

Thanks for any advice.

The original post: /r/datahoarder by /u/brimur on 2024-06-18 14:30:37.

I wanted to get a 4TB drive to add to an existing array or 4TB drives. While searching I came across these two and both are CMR NAS drives but I am wondering why one is much cheaper than the other. I tried Googling ST4000NEZ01 but could not find it

The original post: /r/datahoarder by /u/TheLostWanderer47 on 2024-06-18 13:22:23.

The original post: /r/datahoarder by /u/CoolstarLikesHentai on 2024-06-18 06:35:38.

I am trying to get the original video file from a YouTube video by using the yt-dlp -g command (short for yt-dlp --get-url) to get the video URL and audio URL. I can successfully get both URLs, but when I download them it downloads very very slow (~150 kb/s). Is there any way I can speed up the download speed?

The original post: /r/datahoarder by /u/ElonTastical on 2024-06-18 05:56:04.

Let's say you wanna download multiple links from keep2share, but in that site it is limited by single download, you have to wait two hours before you can download another file. Is there a way to bypass this like the idea I mentioned in the title?

The original post: /r/datahoarder by /u/SbM_Yggdrassil on 2024-06-18 05:48:10.

Hi all, I'm at that part of my homelab journey where I've setup a bunch of fun stuff and now I'm starting to think about helpful stuff like backups.

I have a raspberry pi 4 setup with open media vault and docker running various services. I would like to add one or more services (in containers if necessary) to accomplish the following:

1. Make an image of bootdrives of my computers on a schedule
   1. save those images to the attached storage.
   2. reimaging solution for recovery
2. Backup secondary, non-bootdrives of my computers (just copying is probably fine)

I'm just wondering how to best add services and which would do it. If I should use syncthing/duplicati + something else or if there is one thing that can do it all. I'm not sure how incremental backups fit in here either but I'd like to implement that to reduce the burden of network traffic (in my home we have to use wi-fi a lot).

For drive images I've used the free version of Macrium reflect before (and I've heard of acronis), but just on one of my windows pc's. I'd like to have something scheduled from the server side for centralised management.

Does it make sense what I'm trying to achieve?

The original post: /r/datahoarder by /u/GreenPRanger on 2024-06-18 05:14:29.

The original post: /r/datahoarder by /u/SystemElegant2703 on 2024-06-18 01:24:04.

Is it necessary to purchase a NAS if all I'm really interested in are the data integrity features (i.e automatic hash checking/recording, file self-healing, datascrubbing, etc.)? Currently I use MultiPar and backup my data to M-discs. However, I would like greater certainty that the hashes are accurate. For instance, I got a trojan recently and now I'm left questioning if the hundreds of files I've downloaded since my last backup have any data corruption, silent or otherwise. Since it's impossible to know without the features I listed previously, should I consider purchasing a NAS or is there a method I haven't thought of to ensure the same level of data integrity?

The original post: /r/datahoarder by /u/countesscranberry on 2024-06-17 23:05:44.

No idea where to post this, sorry!

I have a list of about 5,000 addresses. For each one, I want to know the census tract, the voting districts, the region (as defined by my city), and maybe more later on.

How can I set something up where I can match my list of addresses with a list of all addresses in my state (Ohio), cross-reference all of that other data, and have all of that information spit out for each address for me?

Really any way to make this process faster would be appreciated. I’ve found some files online from various government agencies but I’m not sure if they are all relevant or useful. What kind of file types am I looking for? I have some maps overlayed in Google Earth so I can look up addresses and find the information that way, but I’m not doing it one by one. I chatted with my IT guy but he’s part time and didn’t have any standout ideas at the time.

Thank you!

The original post: /r/datahoarder by /u/OverqualifiedTech353 on 2024-06-17 22:38:12.

Clear & open your minds - this one's a bit of a doozie...

So I've recently begun ROM collecting (might as well reveal that since my account is tied to it), and I'm faced with the best way to preserve this data, as well as use it infrequently. Currently the plan is sort of like a 2.5-2-0.5 method:

• Leave a copy of smaller ROM sets on my desktop SSD (non-OS), but move everything else:
  • AND Offload collections (such as handhelds) to 2 or 4TB 3.5" HDDs depending on the collection size and keep in "lukewarm" storage (plug into HDD dock for small updates, otherwise keep offline) [1]
    • AND from there, copy organized folders to accessible micro SD cards for the various handhelds (no power most of the time) [2a]
    • XOR copy organized folders to portable HDDs for the various consoles (off/idle, or online for maybe max 1khrs/yr) [2b]
  • AND Offload copy of entire collection to a large NAS. [3]

I have 6x 2TB HDDs, and 8x 4TB HDDs inherited from various places for the [1] and [3] tasks. So:

[3] For NAS, I am thinking 4TBx6 (-1) = 20TB in RAID5. This would be used for other files and backups as well (that's it's own problem).

[1] That leaves 2x6TB + 2x4TB = 22TB of curated backup/cold storage. Though I am thinking I leave one or both of the 4TBs for replacement drives in case of failure... so 12TB of 2TB drives.

That's sufficient for 2x copies (one online [3], one offline [1]) + the 'accessible'/usable copies [2] (kind of satisfying the "3" and 2 in 3-2-1). The 1 is a 0.5 since it would be in another room(s), and doing a truely remote location is not really in the picture for me. Really all 3 mediums would be in different rooms, likely.

My biggest concern is maintaining the 2 (well, 3) cold storage media types. Especially the microSDs, which I cannot find a definitive answer on how to maintain file integrity (seems like power on at least every year?, then somehow do a bit-by-bit rewrite?)

I've read up on store type archives, and PAR2 type parity. It seems like the best solution for long-term storage for a standalone, non-RAID drive (SD or HDD), would be to create a 10% parity file for every game (zip multi-file games), particularly on the microSDs which are most prone to bit rot.

I've had a lot of issues with microSDs in the past failing, so I have little faith in them. That means I would be forced to keep a per-card mirror on the NAS most likely (in place of the compressed versions for transfer convenience), and backing up any save games at least annually. But then what? Just wait until I try to access a file that fails? And what about the portable and internal HDDs? How often should I be plugging them back in and doing X?

tl;dr: How do you maintain microSDs and offline HDDs, how do you validate them, how often do you clone and replace them? And otherwise what would you do differently or add?

The original post: /r/datahoarder by /u/CreativeDog2024 on 2024-06-17 19:45:30.

I have about 5TB of movies/tv shows/photos. I have 3 backups: one is my main frequently accessed HDD. Another is another HDD i keep in my drawer and the last is google drive.

I'm not a programmer so idk how to verify whether there has been data loss but I make sure all the files on each are the same by using freefilesync (mac app). It takes so much time and I don't even know if the files are corrupted or not.

Is there some cloud option that I can leave my data on and pay $10-15/month to forget about it, using it as a last resort if all my local backups are corrupted?

I read quite a bit on this and people recommend backblaze (B2 I think, how do i even buy it?), AWS glacier and M-disks.

I have no idea how to operate any of those because I don't code. I do use rclone for Gdrive though.