The original post: /r/datahoarder by /u/Leading-Geologist-39 on 2025-01-13 18:37:02.
Found an older cheap consumer Z370 mainboard with a TB header from times where they handed those out on the Intel platforms like candy. Added a cheap same brand TB3 PCIe card (Asus ThunderboltEX 3) with USB-C. It was an old pre-2020 build with an 8700 with a cooler and RAM still on it. Booted linux off a USB drive, installed Samba and plugged in my laptop that has a TB3 USB-C port. Gave the TB3 network link some private IP address on each side and tested performance over that Thunderbolt 10GbE link.
The maximum I could get out of this old Z370 platform with few PCIe lanes was 5 NVMe M.2 drives for a RAIDz1 config. Used 5x2TB Crucial T500 and reading and writing large files from the laptop's very fast NVMe maxes out the link continuously at 1.25GB/s.
Even if you just add a couple old HDDs with a RAIDz1/2 transferring large files will still be around 500-800MB/s easily. The SSDs are definitely a bit wasted due to the 10GbE limitation of Thunderbolt 3.
(I set ashift to 14 to match the 16KB SSD page size spec, scrubs only at 2.5GB/s. I am certain it's entirely limited by the overall PCIe lane constraints, these are old PCIe 3.0 slots where the 2 onboard M.2 SSDs share 4 PCIe 3.0 lanes which is just sad for a modern PCIe 4.0 SSD...)
Plugged in a Macbook with USB-C for fun and ran a Timemachine backup over SMB (with fruit plugin) just to see how stable it is for a variety of workloads. Came back about 20 minutes later to see the 1TB backup already completed. No hiccups, no disconnects, and even if you accidentally unplug it the file system remains unaffected. At most you gotta restart any transfers already in progress.
An actual DAS that doesn't run its own OS with ZFS and just exposes the drives over USB3 or something can certainly run these SSDs at higher speeds than 10GbE but I don't wanna deal with "safely ejecting" and whatnot, and I already had the mainboard sitting in storage ever since I switched to AM4.
The best part is that I can ZFS send/receive between my main storage server (good for backups) and as that runs with snapshots over the regular NIC on the mainboard it's entirely independent of whether I have the Thunderbolt cable plugged into any computer.
Does it have ECC RAM? No. Is it power efficient? With 30W 24/7 not particularly. But my main storage server is sitting in its rack far away and all I get is 1Gbps wired or at best around the same over Wifi to a laptop. Browsing my 4k prores video files where each one is 100GB+ in size that connection is just no longer sufficient in 2025. Having a couple TB of fast temporary space available at my workspace is sweet.
If I had instead added some SSDs to my desktop computer I wouldn't have had a fun project setting up another server with ZFS. This was at first an experiment to see how stable a TB3 direct link really is and now it's a part of my workflow as I found it to be more reliable than any actual DAS I had sitting on my desk in the past.
With a very expensive SFX build and M.2 SSDs you could definitely get the footprint on your desk down to actual DAS size, but I have a 6 foot TB3 cable so I could put the cheap bulky computer case in a corner under the desk where I don't have to look at it.