I don't think compiler optimizations matter much - supposedly the final build was compiled without optimizations, presumably by mistake, and the N64 has very specific hardware which compilers don't know how to optimize for.
What we certainly do have are much more powerful machines and software in general, letting you test, analyze and profile code much more easily, as well as vast amounts of freely available information online - I can't really imagine how they did it back then.
I don't think that's a valid criticism, I don't see death saying that, it'd be more like "Your time has come" - to contrast it further, wouldn't somebody say "It's your time to shine"?