Proxmox

An Intel N100/150/200/250 based mini PC is a really low power (as in energy consumption) platform that is pretty good for PVE. Mini PCs based on the N100 tend to be inexpensive and a good alternative to the ones mentioned in other comments. The power draw is usually between 5-15W - depending on the load

The cheapest i5 or i7 with 16GB of DDR3 and an whatever SSD (bcs 50~100 monies for best components of the era, they can idle ok-ish, have SATAs, prob even 5.25" bays for them nice rubber 3.5" HDD adapters since it's provable that you plan to have the PC in/close to your living space).

However have in mind that you very probably have to buy at least one peace of nice storage, such as at least one or two enterprise HDDs (I would say one & plan on getting up a second PC eventually for full redundancy/copy & skip local/raid redundancies).
So that's 100~500 monies maybe, but this already depends on your intended use (or just play with desktop HDDs for a while but they don't last, also smol).

And you might wan't to buy an efficient & nicely built gold PSU (I have mostly used the better Seasonics, but a list like this can help you get you started). Desktops don't usually have PSU for 24/7, might not be safe, and use a lot of power (especially when idling they can have horrible efficiency & affect your electricity consumption).
So another 100 monies but you could get a used one too.

PC is the way to go imho, for a small amount of concurrent uses they are way faster than serves for the same price (you don't need extra throughoutputs, PCIe lanes, or closed sauce megacorp server tools), are very loud, and consume a lot of power (especially old ones don't even really idle/drop the voltages on CPUs).

Good recommendations already, but to get started just use literally anything. Any x64 Dualcore with 4GB RAM is sufficient to just try stuff, even though you'll reach the memory limits sooner or later. You can poke around with it and figure out what you actually need.

Check eBay for used enterprise machines. Dell optiplex 7050, Lenovo m75, HP elitedesk 705 or 800. All pretty cheap and have availability in the mini format. Find an Intel x550 card for 10gbe if needed.

I started with a Dell Wyze 5070 from eBay. I put a 1Tb M2 sata in it, and 32Gb of RAM. Be warned it's really picky about RAM type. After a while I added 2 more, for testing and to spread the load a bit. I also added a proxmox backup server. I like the little boxes as they run at about 15W each.

If I was doing it again, I'd try and find a lowish power box that wasn't so picky on RAM, and also look at something with a 10GbE port so I could run ceph for some shared storage - but that pushed the price up...

HTH

If you're running containers for things that don't require a lot of resources, you can do a lot with a used NUC or micro form factor PC with i5 or Ryzen 5 from the 2020s, 8 GB RAM, and a 256GB SSD. You can get business desktops on eBay and elsewhere for pretty cheap and the i5/8GB/256GB or R5/8GB/256GB configuration is everywhere. If you want, you can even pick a machine that has a single 8GB stick and add a second, or you can replace the storage with a larger drive. But either way, cruise used listings for off-lease or recently upgraded business gear and then grab 3rd party RAM/storage to upgrade as needed from there. I grab Ryzen when I need stuff and it's available because it usually saves a few bucks and works great. More and more businesses are buying systems with 16GB RAM as a baseline so you might find one of those machines for cheap and you just doubled your memory.

I've got 3 NUCs just like this set up along with a dedicated NAS to act as a storage repo.

If, instead of containers, you are running full-on virtual machines, that changes things because the VMs will allocate whatever resources you allow them to have access to like a greedy asshole roommate. I really like using containers for the stuff I keep in Proxmox though.