I bought a 16-slot LTO-4 changer a while ago and had gone back and forth on whether or not I wanted to use a proprietary software to do my backups.
I decided that I didn't - so I had to figure out how to get things to work reliably for my archives. Specifically I wanted to use the LTO drive to create cold offline and offsite copy of critical media. Please assume for the remainder of this post that I have ascribed to whatever onsite/offsite dogma that you like and value, and that you and I agree 100% on all my calculations/valuations of cost/value/time in regards to this endeavor.
I have a 75tb Synology NAS and a Proxmox server thats running all my household stuff, as well as hosting the Ubuntu instance that my LTO drive is connected to. I dont have enough local disk space on my server to host the data I want to backup locally, I need to take it straight from the NAS.
Sooo...I started with trying to stream tar across the network. like this:
serverwithtapedrive# ssh user@nas.device.local "tar --tape-length=810000000 -cMf - '/volume1/FolderToBackup'" /dev/nst0
Straight tar streaming across the network.
If you have only big files - (mostly) no problem. Small files (images in my case) wreak havoc on the throughput. This quickly made it impossible for me to feed the LTO drive fast enough and was a failboat.
The interwebs told me that I could use mbuffer to create a buffer in memory that I could write to on the tape server from the remote. This sounded great so I tried it like this:
serverwithtapedrive# ssh user@nas.device.local "tar --tape-length=810000000 -cMf - --blocking-factor=512 '/volume1/FolderToBackup'" | mbuffer -m 4G -P 80 -s 512k -t -o /dev/nst1
And this seemed good-ish. This creates a buffer in memory on the server (4gig in this case) and when it hits 80% (-P 80) it starts dequeuing data (the tar) into the specified tape device. And this definitely smoothed out the network transport - but I still couldnt keep up with the write speed of the tape - even with a 50 or 100GB queue. Id fill it up to 80%, it would start to dequeue at 100-120MB/sec (LTO-4's throughput) and empty the queue, putting me back into underrun. Damnit.
So after quite a bit of machination - I ended up splitting the operation into server (where the tape drive is) and client (where the data is that I want to backup). And here's what it looks like:
First on the server - make sure you have a tape loaded into your device (mine is /dev/nst1) and execute the following:
server# mbuffer -I 8000 -m 200G -P 80 -s 512k | pv -L 40M > /dev/null 2>&1 | dd of=/dev/nst1 bs=512k iflag=fullblock status=none
This sets up mbuffer to listen on port 8000/tcp, creates a 200G memory buffer (I have alot of ram, just not alot of disk space - you dont need 200GB, experiment with sizes to see what works for your network), dequeues when the buffer hits 80%, uses a 512k blocksize THEN toss the data into pv - where we limit the dequeue to ~40Megabytes/sec (this is ~10megabytes/sec above the lower bound of safety for LTO-4 - this is how I ended up solving the buffer underrun) then throw the data to dd to block if up and chuck it into the tape drive.
Then on the client:
client# tar -cMf - --tape-length=1647820800 /volume1/FolderToBackup 2>/dev/null | pv -s 3622336905216 | nc tape.server.ip.address 8000
Tar up (specifying the length of my tape as 790G ) , pipe it to over to pv where I told it my total backup size (3.3TBish) so it will give me a rough eta and progress bar, and then pipe it to nc to shoot it over the network to the server.
And...? It works. I have rock solid queue depth and dequeue on the server, and I can backup effectively, my large sets to my tape changer.
I wrote this wall of text hoping that it might save someone else from spending as much time as I did trying to sort out how to make a cheap old perfectly good tape drive do work over the network.
