Luke Kemp, a research associate at the Centre for the Study of Existential Risk at the University of Cambridge, has written a book about his research called 'Goliath’s Curse: The History and Future of Societal Collapse'.
He makes the case that, from looking at the archaeological record, when many societies collapse, most people end up better off afterward. For example, people in the post-Roman world were taller and healthier. Collapse can be a redistribution of resources and power, not just chaos.
For most of human history, humans lived as nomadic egalitarian bands, with low violence and high mobility. Threats (disease, war, economic precarity) push populations toward authoritarian leaders. The resulting rise in inequality from that sets off a cycle that will end in collapse. Furthermore, he argues we are living in the late stages of such a cycle now. He says "the threat is from leaders who are 'walking versions of the dark triad' – narcissism, psychopathy and Machiavellianism – in a world menaced by the climate crisis, nuclear weapons, artificial intelligence and killer robots."
Some people hope/think we are destined for a future of Universal Basic Income and fully automated luxury communism. Perhaps that's the egalitarianism that emerges after our own collapse? If so, I hope the collapse bit is short and we get to the egalitarian bit ASAP.
Collapse for the 99% | Luke Kemp; What really happens when Goliaths fall
Yes, modern yields could not be sustained for long, that's true. If we assume whatever killed civilisation didn't kill a lot of people already, famine is guaranteed. You made it sound like everyone would stay put, though.