It's A Digital Disease!

The original post: /r/datahoarder by /u/TraditionalTask9945 on 2024-09-27 13:44:17.

(beginner, don´t know anything, seeking advice)

USB sticks backup: Documents, pictures, simple stuff to backup.

Whenever new files are added, I copy the folder with the new stuff, and paste that to the the backups with the folder without the new stuff. Get prompted do you want to write over these files. Is this the way to properly backup?

Is there a software that only adds the new files, but skips the files that are the same, and confirms that all backups are now updated and identical?

Tried using robocopy but there is something wonky going on where usb "disconnnects" constantly during the process and robocopy says it has to wait for 30 seconds. But every usb port has been switched permanently on, in the battery savings settings.

The original post: /r/datahoarder by /u/xSickBoyx on 2024-09-27 12:50:48.

I've set up a 6-bay Ugreen NAS with unraid as OS (planning on 4 drive pool + 2 parity) as was looking for a simple way of backing up the data regularly to an external device or enclosure. After research, I settled on the OWC Thuderbay 8 but then realized the price of the empty enclosure $749 (don't wanna deal with SoftRaid) is almost the price of another Synology or another Ugreen NAS.

I'm essentially looking for a 6 bay enclosure I can connect via USB / TB and schedule a full NAS backup for disaster recovery but $800 seems pretty pricey for a box with no CPU or software. Any recommendations?

The original post: /r/datahoarder by /u/rohithkumarsp on 2024-09-27 12:19:47.

The original post: /r/datahoarder by /u/Loitering14 on 2024-09-27 11:35:10.

The original post: /r/datahoarder by /u/SAA2000 on 2024-09-27 09:33:02.

Site admins estimate size to be around 2.7TB with 100GB per-month growth.

Link: https://info.arxiv.org/help/bulk_data_s3.html

The original post: /r/datahoarder by /u/IShunpoYourFace on 2024-09-27 09:06:34.

I have WD40EFZX and its quite noisy while browsing thumbnails(idk if its failing, been running for 15k hours, 98 power on counts, 17k load/unload cycles). I'd like to upgrade by buying one drive for backup thats going to be mostly in sleep (spun up once a day for something like volume shadow copy) or off if its USB DAS drive, should I go with shuckable drive or 3.5" nas drive? I'm trying to keep 4TB of important data on at least 2 copies until I can afford more storage, ecc ram and zfs in something like raid 6 with one offsite backup.

The original post: /r/datahoarder by /u/MBaggott on 2024-09-27 03:37:54.

Years ago, I worked in a place that I believe ran on some version of Windows 97, which even then was ancient. Long story short, I have an encrypted Windows97 hard drive that has my old files backed up on it, but I can't access them because the thing is encrypted. Is there any chance that decrypting these things has become an easy task since then?

The original post: /r/datahoarder by /u/cassutti on 2024-09-27 08:19:14.

The original post: /r/datahoarder by /u/UnappreciatedGraf on 2024-09-27 07:16:54.

I'm planning to buy a 4tb ssd but have a few questions. The thing is, intend to use it exclusively externally in an enclosure (for the fore seeable future at least). Most of the buying guides that I see on yt are made with the intent of using the SSD internally, because that's the more common use case ofcourse. My question is, that for MY external use, do DRAM yes/no; TLC/QLC; TBW rating even matter?

ALSO, the ssd's I'm eyeing rn are said to have a theoretical max speed of somewhere in the range of 6gb/s. Now, all enclosures online offer USB 3.1 gen 2 which implies 10gb/s. Even tho my laptop rn has 3.1 gen 1x1, i.e 5gb/s max, if I upgrade it in the future to be compatible with latest USB protocols, will I get anywhere near that theoretical 6-7gb/s speed? if not, what will the bottleneck be? The enclosure, the USB protocol or my Laptop?

The original post: /r/datahoarder by /u/ptoki on 2024-09-27 04:45:52.

I am recording two radio stations for almost 3 years now.

I have those duct tape and wd40 scripts to do that:

# cat /root/nagraj_radio.sh
#!/bin/sh
cd /mnt/storage/record_radio
timestamp=`date +%Y%m%d-%H%M%S`
echo "------------- $timestamp -------------------"
mplayer -dumpstream -dumpfile "$timestamp radio.mp3"  -fs ffmpeg://https://an02.cdn.eurozet.pl/ant-kat.mp3 2>&1 >>   radio_record.log &
PID="$!"
sleep "60m"
kill "${PID}"
sleep 5
kill -9 "${PID}"

Run this every hour from crontab.

It works on my debian/ubuntu for almost 3years now and is pretty stable. It fails only if my internet drops for multiple minutes or power outage. The buffering usually takes care of the rest. I listened to a number of those clips and they are always complete and are 1h15s long so there are no gaps.

Usually those create 70-90MB worth of data per hour.

I use it to listen to that radio in my car and to be able to rewind, pull some mp3 songs out of there and listen to some programs I like offline or when they are no longer aired.

Feel free to reuse after changing the station url.

The original post: /r/datahoarder by /u/StuntGuy on 2024-09-27 01:54:44.

After installing my new red plus 10tb hdd I started to transfer over 6 tbs worth of my files from the previous drive and about 4 or 5 tb into it I randomly decided to touch the drive bay and I was shocked at how hot not even the drive was but just the metal drive bay case it was in! I then touched the back of it the part that is exposed and it was so damn hot like felt like almost burning me hot!!!! I decided to just press pause on the file transfers for now but it's been about 30 minutes on pause and it still seems pretty hot.... is this normal? I'm not so sure because I touched the older wd black drive that I was getting that big 6tbs from and it should be hot as well no because it transfered that entire time too but it wasn't hot at all!!?? Maybe writing files makes it hotter than it would be coppying? Please any advice is appreciated

The original post: /r/datahoarder by /u/Tobobo4899 on 2024-09-27 01:35:35.

I'm going to be upgrading my whole pc setup this weekend. My plan is to install my new os drive , download steam, download a small game in my separate games ssd drive to build the file structure and then copy the contents of my steam apps - common folder from my old HDD to this new file structure on the new drive.

I'm then going to remove the HDD, and in steam download each of the games I've copied, I'm told it will verify and install any files that weren't copied over. Then I can run the games and select my steam cloud backup to reinstall my saves. (I have manual backups of saves too)

My question is will this work? I'm stressing myself out over analyzing all the steps trying to make sure it will work.

The original post: /r/datahoarder by /u/phul_colons on 2024-09-27 00:13:15.

Current setup:

• 8x16TB raidz2 on Ubuntu Server
• 6x8TB, 4x10TB Drivepool on Windows 10

I want a 3rd version directly attached to my M1 Max Macbook. I have 2x14TB, 12TB, 2x8TB to work with and a 5 bay Sabrent USB-C DAS.

I've tried

• open ZFS on macOS. I get a kernel panic and sudden restart as soon as I start heavy IO. The whole system becomes fairly unusable before that panic occurs. Single digit refresh rate on the screen, no input from keyboard/trackpad. I tried with raw disk partitions and using sparse files stored on APFS/case sensitive/encrypted containers.
• APFS containers with multiple disks. There's no redundancy available. No performance benefits to pooling multiple disks together. No replacement strategy for failed disks.
• DiskUtility software RAID with JBOD. Disks are filled one-at-a-time and the dimension cannot be modified after it's created to allow for expansion. Disks cannot be read independent of their grouping. One disk failure and you lose the whole thing. Zero performance benefit on reads or writes.
• Standard APFS with some kind of disk pooling strategy. MergerFS does not work on macOS. There is no native pooling option either. symlinks don't work with Syncthing, which I'm using to keep all 3 systems in sync.
• Passing the disks to a VM, using ZFS, then sharing back over SMB. It is simply too cumbersome, especially with the apple silicon architecture. Nothing reliably works and even test scenarios are painstakingly difficult to set up. Virtualization seems to have taken a decade step backwards on apple silicon.
• Docker desktop. No option on mac/Windows to pass USB devices like you can on linux.

What I want in descending priority order:

• Disk pooling so I can sync my 20TB folder
• Options for redundancy, either mirroring or striping.
• Options for replacement of failed disks.
• Options for expansion.
• Performance benefits afforded by using multiple disks.

Encryption is a requirement, but I'm not willing to use file-level encryption like encfs or similiar. The only thing I haven't tried yet is SnapRAID, but if disk pooling doesn't work then I'm not interested at all.

I'm at my wits' end with this. I'm convinced macOS simply has no options available for multi-disk storage needs. Can anyone share something that has worked for them?

edit: I'm experimenting now with the native RAID assistant, making two JBOD arrays and then mirroring them to form a 28TB mirrored drive at the end (14/14 TB JBOD mirrored with 12/8/8 TB JBOD)

edit2: So I've currently got this setup going:

(14-14 TB JBOD) mirrored with (12-8-8 TB JBOD) APFS, case-sensitive, encrypted container

Write speeds: 150 MB/sec Read speeds: 200 MB/sec

This may be workable, I'll test it for a while and see how far it gets me. Thanks for reading.

edit 3: settling on 14-14 TB raid0 + rsync to 12-8-8 TB jbod using the macOS RAID assistant.

Write: 450 MB/sec Read: 400 MB/sec

This gives me great local performance + a full backup. 28TB usable space.

The original post: /r/datahoarder by /u/ghinghis_dong on 2024-09-27 00:02:36.

I’ve noticed that this form factor is starting to show up as used drives on eBay. They are petty cheap relative to U.2/U.3 form favors for the same capacity (micron 7450 u.3 vs E1.S)

However, I don’t see any cheap way to use them. For example, an add in card for PCI or some kind arrangement with fiber channel. In fact, searching for servers that use this hasn’t really tuned up much that I would realistically buy. This is harder than even U.2…

Any thoughts?

The original post: /r/datahoarder by /u/Man_who_says_heIlo on 2024-09-26 22:36:53.

I have an account where I send myself stuff that I want to download and I wonder if there is a way to download all of it without going through it one by one

The original post: /r/datahoarder by /u/angry_dingo on 2024-09-26 22:20:09.

SpinRite has been significantly improved

After 20 years, SpinRite 6.0 has been updated to 6.1, and as a licensed owner of 6.0, you are invited to upgrade your copy of SpinRite at no cost.

SpinRite v6.1 is (finally) blindingly fast and

SpinRite restores SSD performance

The biggest surprise was discovering that even solid state mass storage could benefit from SpinRite's rewriting of its media. Due to “read disturb”, SSD and Flash storage slow down when they are only read and never written -- which affects most operating system files. People say that their SSD-based laptops no longer perform as well as they once did -- now we know why.

During the 3.5 years of work on v6.1, we discovered that SpinRite could restore SSD original factory performance.

There's MUCH more to say and share. You can find everything at grc.com.

I'm stunned. I bought my copy 20 years ago. This was great software a long time ago and I thought it would never be updated. Can't wait to try it out.

The original post: /r/datahoarder by /u/OliDouche on 2024-09-26 21:41:06.

Hi friends,

I could use some help with evolving my setup to the next level!

Current Setup

• Initial Configuration:

  • Started with a Mac Mini.
  • Purchased OWC enclosures (connected via Thunderbolt) and filled them with drives.

• Transition to Windows:

  • Moved to a Windows host using the same enclosures.
  • Struggling with Windows’ RAID handling capabilities.

• Current Software RAID:

  • Using OWC's SoftRAID for software RAID.
  • Hosting a variety of services:
  • Jellyfin for media
  • Immich for photo backups
  • Mostly Dockerized containers and some nodeJS applications
  • All data is stored in the OWC enclosures.

What I Need Help With

I want to move away from this setup to something more appropriate. Here are some details about my existing server:

• Custom Built PC (similar to a gaming PC)
• Specs:
  • Drives: 18TB x 8, 16TB x 8
  • CPU: Intel 13700k
  • RAM: 64GB 6400MHz
  • GPU: Intel ARC A380 (for HWA and AV1 encoding)

Issues:

• Windows is problematic with RAID setups.
• I don't want to rely on external drive bays (especially via Thunderbolt) for my data.

What I’m Looking For

Suggestions on transitioning to a more reliable and integrated storage and server solution. This is my first time venturing outside of a PC+DAS server setup, so your help (and patience!) is greatly appreciated. I'm really curious about getting started with a rack chassis - but it's all very intimidating and I could use some help as to where to look.

Thank you!

The original post: /r/datahoarder by /u/LouieVbbp on 2024-09-26 21:23:45.

The original post: /r/datahoarder by /u/MrWoohoo on 2024-09-26 19:30:02.

All of the software I've found so far will search and return saying it found a number of duplicates. If I ask the software to delete duplicates the program will randomly choose one to keep.

What I would like to mark a folder as the master then have the program delete any copies NOT IN THE MASTER DIRECTORY. Anyone know of such a beast?

Thanks.

The original post: /r/datahoarder by /u/ptoki on 2024-09-26 19:28:15.

Hello. I need to get my drives to a better config. Currently they run over usb and they tend to vanish occasionally and only powercycle or remount helps.

I am looking for a decent solution to hook up 4 sata drives and have it semi decent.

I saw some folks finding older microservers from hp, some others use fancy sata splitters, others use synology/qnap solutions.

But I cant find them chaeap. That is for less than 200USD.

I figured out that older computer (more like 10 year old) I have and can repurpose will do the trick but I wonder if there is anything I missed.

Any suggestions?

The original post: /r/datahoarder by /u/oShievy on 2024-09-26 18:20:13.

I have a DIY NAS build that I made about a week ago. After reading several things about data corruption on ZFS (currently using proxmox for this) via non-ecc ram and power outages, I figure if I spent the time and money, I may as well ensure everything will work as well as possible.

However, I live in an apartment and am unable to do things as efficiently due to space. Currently I have this power strip and plan on getting this UPS, then plug the power strip that my NAS is connected to in the UPS.

My average load does not exceed 70w. I think the maximum power draw id ever get close to achieving would be maybe 150-200w. I am only plugging this NAS into the UPS as well, no other devices.

Is this advisable? Or even necessary really? I have seen online people able to recover their pools even after power loss, and after reading it seems that ZFS is more resilient to power loss due to its CoW functionality.

Another (maybe safer?) option would be to get a longer power cable (around 7ft) so that it can reach the UPS at the wall. Wouldn't have to deal with any power extension cords/surge protectors here, just the UPS. Let me know what you guys think.

The original post: /r/datahoarder by /u/SpaceNut1976 on 2024-09-26 18:06:48.

Trying to save off a video and I have exhausted my resources. Tried numerous online and command line tools and I’m stuck.

https://www.ultimedia.com/default/index/videogeneric/id/x0um38q

The original post: /r/datahoarder by /u/humor4fun on 2024-09-26 16:28:27.

yoto-archival-downloader

I forked this from another person who did most of the heavy lifting. After finding it, I expanded it significantly with a bunch of new features designed for creating archival replicas of the contents. I welcome anyone's feedback about it. I'd love some help turning this into a tool that can just be fed a URL and have the tool fetch the files, then zip them up. eventually I might get there. For now... this is where it is at.

Note: I am not a JS programmer by trade these days, just some hobby work.

The original post: /r/datahoarder by /u/datawh0rder on 2024-09-26 16:17:16.

So I recently purchased a 24TB HDD to back up a bunch of my disparate data in one place, with plans to back that HDD up to the cloud. One of the drives I want to back up is my 2TB SSD that I use as my Time Machine Drive for my Mac (with encrypted backups, btw. this will be an important detail later). However, I quickly learned that Apple really does not want you copying data from a Time Machine Drive elsewhere, especially with the new APFS format. But I thought: it's all just 1s and 0s, right? If I can literally copy all the bits somewhere else, surely I'd be able to copy them back and my computer wouldn't know the difference.

Enter dd.

For those who don't know, dd is a command line tool that does exactly that. Not only can it make bitwise copies, but you don't have to write the copy to another drive, you can write the copy into an image file, which was perfect for my use case. So I used the following commands to copy my TM drive to my backup drive:

diskutil list # find the number of the time machine disk

dd if=/dev/diskX (time machine drive) | pv | of=/Volumes/MyBackupHDD/time_machine.img

This created the copy onto my backup HDD. Then I attempted a restore:

dd if=/Volumes/MyBackupHDD/time_machine.img | pv | dd of=/dev/diskX (time machine drive)

I let it do it's thing, and voila! Pretty much immediately after it finished, my mac detected the newly written Time Machine Drive and asked me for my encryption password! I entered it, it unlocked and mounted normally, and I checked on my volume and my latest backups were all there on the drive, just as they had been before I did this whole process.

Now, for a few notes for anyone who wants to attempt this:

1. First and foremost, use this method at your own risk. The fact that I had to do all this to backup my drive should let you know that Apple does not want you doing this, and you may potentially corrupt your drive even if you follow the commands and these notes to a T.

2. This worked even with an encrypted drive, so I assume it would work fine with an unencrypted drive as well— again, its a literal bitwise copy.

3. IF YOU READ NOTHING ELSE READ THIS NOTE: When finding the disk to write to, you MUST use the DISK ITSELF, NOT THE TIME MACHINE VOLUME THAT IT CONTAINS!!!! When apple formats the disk to use for Time Machine, it's also writing information about the GUID Partition Scheme and things to the EFI boot partition. If you do not also copy those bits over, you may or may not run into issues with addressing and such (I have not tested this, but I didn't want to take the chance. So just copy the disk in its entirety to be safe.)

4. You will need to run this as root/superuser (i.e., using sudo for your commands). Because I piped to pv (this is optional but will give you progress on how much data has been written), I ended up using "sudo -i" before my commands to switch to root user so I wouldn't run into any weirdness using sudo for multiple commands.

5. When restoring, you may run into a "Resource busy" error. If this happens, use the following command: "diskutil unmountDisk /dev/diskX" where diskX is your Time Machine drive. This will unmount ALL volumes and free the resource so you can write to it freely.

6. This method is extremely fragile and was only tested for creating and restoring images to a drive of the same size as the original (in fact, it may even only work for the same model of drive, or even only the same physical drive itself if there are tiny capacity differences between different drives of the same model). If I wanted to, say, expand my Time Machine Drive by upgrading from a 2TB to a 4TB, I'm not so sure how that would work given the nature of dd. This is because dd also copies over free space, because it knows nothing of the nature of the data it copies. Therefore there may be differences in the format and size of partition maps and EFI boot volumes on a drive of a different size, plus there will be more bits "unanswered for" because the larger drive has extra space, in which case this method might no longer work.

Aaaaaaaaand that's all folks! Happy backing up, feel free to leave any questions in the comments and I will try to respond.

The original post: /r/datahoarder by /u/Daconby on 2024-09-26 15:53:24.

Looking at specs of older, data center quality expansion cabinets from companies such as Dell and HP, I see large current draw numbers, such as 7A at 120V for a 16 bay cabinet. Assuming each drive uses 10 watts max, that's only 160W. Add a few fans and 10% loss for power supply inefficiency, and you're still only looking at maybe 2-3A. Watt's going on here?

To be clear, I'm not talking about entire servers, these are only drive cabinets.