There's historically been some privilege escalations, such as https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-32233
But at the same time, they do offer increased security when they work correctly. It's like saying we shouldn't use virtualization anymore because historically some virtual devices have been exploitable in a way that you could escape the VM. Or lately, Spectre/Meltdown. Or a bit of an older one, Rowhammer.
Sometimes, security measures open a hole while closing many others. That's how software works unfortunately, especially in something as complex as the Linux kernel.
Using namespaces and keeping your system up to date is the best you can do as a user. Or maybe add a layer of VM. But no solution is foolproof, if you really need that much security use multiple devices, ideally airgapped ones whenever possible.