RetroGaming

I have an Anbernic RG353M & it sounds like it fits your needs. Emulates up to PSP. Moddable: I run ArkOS which is Debian-based - so I can SSH in, apt install programs, run custom bash scripts I wrote, etc. I also ported a desktop game to run on it. So moddability points are high! Also dual-boots into Android, so Android games are available if that's your thing. It comes with the Android dual-boot right out of the box!

Downsides:

• It can be a little small in my hands for long play sessions. I have a custom 3D printed handle extension which works great and solves this problem.
• There's a hardware problem involving the 3.5mm audio jack output where chip noise gets in the audio. Annoying, but ignorable. Switching to Bluetooth headphones is a fine workaround too.
• In long play sessions when the device gets hot horizontal bar artifacts show up on the screen. Like the audio thing, they are annoying but ignorable. When they show up I normally interpret it as a sign I've been playing too long and take a break while the device cools down.

Overall, it's got those hardware quirks above but I still like it and I don't regret it for the price (something like ~$100 a couple of years ago I think). Battery life is very good, even for graphics intensive games. It can go many hours without needing a charge, and generally it has better stamina than I do!