Matrix

Like you said, matrix users are concentrated in one server in a way that XMPP isn't. If one XMPP server introduced this system, it would have a much smaller impact than this change will have on matrix users, so that's a factor in assessing the risk of future problems with the two networks.

Personally, I'm undecided. I used XMPP for years and now I'm using matrix (both self hosted). Neither is perfect. Element has some shortcomings that are addressed in element X, but not all the other features have been migrated. I found the other matrix client I used for a bit, "fluffychat", is better than element but has some other bugs. I end up switching keeping more than one client installed and switching fairly often to work around bugs, which is not ideal.