The structures that unambiguously always define male or female are the structures that produce functional gametes. I interpreted "how folks that were never going to produce either fit into that definition" as asking "If we don't look at the gonads, what would we use to determine sex". Those ducts are a very good indicator, but are secondary structures around the gonads. If you wanted to determine sex without looking at gonads, those are one of the primary structures for doing so.

Ovotestes are interesting, but probably not what you're thinking. They're not just normal testes and ovaries as one might be lead to believe from the name. They're exceedingly rare, so have to be examined individually and general statements can't really be made. You'll probably find a (semi-)functional gonad from which their sex would be determined, with a sampling of non-functioning tissue from the other sex. You'll also likely find that the surrounding structures and rest of their body is unambiguously male or female, though again you'd have to look at a specific case.

To bring it around to near the start of this thread, even then, the body isn't organized around producing no gametes. It's organized around producing gametes and failing to do so.