The core tenet of tribalism is "They aren't like us." That might be based on skin color, hair type, clothing, smell (from different diets), behavior. Modern racism (from the last couple hundred years) likely has some elements of more traditional tribalism with relaxed standards so the people a few hundred miles away can start to wrap their heads around the idea that Irish, for instance, are more or less the same as British.
I do hope people can get to the idea that anyone from a given point on this planet (so far) is just a person and not an outsider, but it looks like we have a way to go.
For the last 200 years, a significant amount of slavery has been limited to certain phenotypes. I agree that prior to that, it was less prevalent. That doesn't mean we don't have a historical model of slavery based on phenotype, it's just more recent history.