I'm not denying that ace spectrum people have relationships and settle down as well. But OP is asking about the normative (read allosexual) experience and explicitly mentions physical attraction.
The vast majority of relationships will involve physical attraction and sex. It's highly unusual for that to not be the case for allosexual people. That's not a value judgement—if a minority of allo people find something else works for them, then that's great. But if OP is asking if this is normal, then no it's not. Even 'less attractive' people, as OP put it, find people they're attracted to enough to enjoy a lifetime of intimacy and sex with.
Overcoming a lack of physical attraction is a pretty big barrier and I can't see most people overcoming that barrier just to 'settle down.' Not being your physical ideal is one thing, most of us settle down with people who don't look like models or actors, but finding someone physically unattractive is a tough sell in most cases.
No one is entitled to sex, I agree. But in a relationship, you are allowed to make clear what your needs are and move on if those needs aren't being met. It's not entitlement to know what you want. Having a 'dead bedroom' is why a lot of relationships end.
If I may ask, was there any physical attraction between you two when you met?