The government and its rules are like Tinkerbell; they only exist if we all believe in them. Or, to quote my boy Montesquieu: governments rule by the consent of the governed. Don't forget that the first major native American relocation program (Andrew Jackson's trail of tears) was ruled unconstitutional. Jackson was a real piece of work and had a fucking hate boner for everyone and everything; I mean, thank fuck the guy wasn't around for WWII and the technology/political ideologies of the time. I'm pretty sure he came when he heard the SCOTUS ruled against him. He famously said "Justice Marshall has made his decision, now let him come enforce it", and just went ahead with his act of genocide. The SCOTUS couldn't really do anything besides stamp its feet and yell about it because Jackson just decided that the rules didn't apply to him.

We've actually been in deeply, deeply unconstitutional territory multiple times in our history and the court either turned a blind eye to it or they were just flat out ignored. It can and will happen again.