Here's the direct comparison:

https://www.hardwaredb.net/geforce-rtx-3070-vs-radeon-rx-6800

Would it be worth it?

If you're in the US, looking at US prices (~$520), hellllllll no.

You can get a 9070 (non xt), for almost the same price (~$550), and thats a much, much better GPU, much more performance per dollar.

https://www.hardwaredb.net/radeon-rx-9070-vs-radeon-rx-6800

Or, you could get a 9060 XT (16 GB) for half about half the price (~$270), and still have a probably overall better card in most situations.

https://www.hardwaredb.net/radeon-rx-9060-xt-vs-radeon-rx-6800

And yes, a 9060 XT (16 GB) beats a 3070.

https://www.hardwaredb.net/radeon-rx-9060-xt-vs-geforce-rtx-3070

Double the VRAM, near half the cost, roughly 20% more gigapixel throughout.

Yeah, older model GPUs, the used market, at least in the US?

Pricing is delusional right now.

Apparently nobody got the memo that new, current AMD GPUs, have actually fallen in price a good deal in the last 6 months.

I've switched from 3090 to the XTX few months back because of a really weird and unique bugs in one of my games (but otherwise it worked alright) and for few months had a lot more problems then I had with nvidia, especially with suspend but sometimes it would just crash (Plasma).

I did also switch to Plasma/Wayland with the upgrade so who knows but few kernels/mesa versions back were really bad, it works nicely now though [Arch]

i switched from rtx 2060 to 5700 xt just a few weeks ago, and a bunch of things run well that barely could before. i was mostly concerned with the forza horizons, they run perfectly now

however there are some issues, mplayer/mpv shows a blue screen on most output drivers, and steamvr just crashes, so there are things to figure out (suggestions are welcome)  

(btw i played w3 for a bit before on nvidia, and i had no issues, just to complicate things)

If things work for you with Nvidia then probably no. But since you mentioned issues, then AMD might help. My gaming box has had an AMD gpu and I don't remember seeing any problems related to it in at least a year. With Wayland, too.

Are you using the official nvidia drivers?

If you're playing Witcher 3 with the RTX on and the next-gen update then it's probably your VRAM limiting you.

https://www.protondb.com/app/292030

Seems to generally work OK with Nvidia, but you may want to try Proton experimental or GE.
If you search 3070 one user also using NVIDIA GeForce RTX 3070 seems to have it running out of the box with only a minor glitch.
But there seems to be a pretty heavy overweight of Radeon users. But IDK if we can draw any conclusions from that?
The game has platinum status, so it ought to work on a decent GPU/system like yours.

I've had no issues with nVidia GPU and TW3, running it with protonGE 9.26. Try that first if you haven't already