Hey y'all, I don't know that this is the best place for my question, but I suspect that folks around here will either have advice or know where to point me if I'm better served asking this somewhere else.
I built out my homelab to learn and get a job in IT. Mission success there. I've since gotten into all kinds of stuff and have a need to rework my storage, but I'm not sure how best to set things up.
Here's what I'm looking to do:
Complete separation of storage and compute
Compute consists of a handful of VMs running on Proxmox, and critically, a Kubernetes cluster
The cluster is currently running in VMs but I have plans to move just about everything into the cluster and move into SFF PCs
This is partially why the complete separation is important, for flexibility and future migration
Here's what I have currently for equipment:
Brocade switch with 48x 1gbe and 8x10gig sfp
2x 12 bay 3.5" servers, one with Proxmox, one with TrueNAS
1x 24 bay 2.5" server
Repurposed NetApp 24x 3.5" chassis, modified to work as DAS
4x 16gb spinners, currently in raidz1
~32 old 2, 3, and 4gb spinners
I found that K8s cluster was very unstable with my first setup, which was an iscsi target on the TrueNAS server. The iSCSI share was mounted in Proxmox with LVM to create K8s VM OS and NFS storage for containers. This got much better when I moved to a local 3x SSD zpool on the Proxmox host. I understand that the latency with spinners and over the network was likely the cause, but this doesn't allow for the separation if like.
Use here is all over - media server is the main thing currently, which for transcoding needs fast seq read/writes, K8s app dev for work, and all kinds of testing containers for work and home. I'd also like to put all VM storage remote as well to play around with some different compute setups. I'd really love to be able to fully wipe Proxmox and try out XCP-ng, or Azure HCI, stuff like that... It's a lab, after all.
I'm competent enough to implement a solution, but I guess not enough experience yet to design. I'm not afraid of complex setup, but I do need redundancy, particularly if putting these small drives to use as I don't trust them - came for free with the NetApp shelf.
So... Any suggestions on how to set this up for minimal latency and fast read/write? Ceph? Mirrored ZFS using all those small spinners? Dedicated ZFS pools for each K8s node? Caching layers? Not against additional equipment, within reason, if needed.