How is your RAM being used? Look at CPU-X in the "System" tab. If the memory is mostly used for buffers and cache then it's not a problem, you want it to be used like that.
I ran my NAS for years off an i5 (Kaby Lake) with 4 GB of RAM and 32 GB of NVMe storage and it handled any of the usual media servers just fine. I've used them all, Plex, Emby and now Jellyfin.
Have a look at this table, get the cheapest used Intel CPU you can find that fits your transcoding requirements, slap it on a board with enough SATA connectors and 4 GB of RAM and you should be good to go.
Docker should not have a large impact, I have 15 containers running right now and they only use 2.5 GB of RAM in total (for reals, without buffers/cache).
Install DroidcamX and turn them into webcams or surveillance cams. It works over LAN and over USB. On LAN you can connect the phones to the server or the server to the phones, as you wish.
On Linux they're seen as regular V4L2 devices. You can use its custom kernel module (installs via DKMS) for additional customization options or the standard V4L2 kernel module.
I'm not really knowledgeable about recording and motion-detection (I've only used it as a webcam) but perhaps someone else can help with that.