It probably did make a difference, even without civilization, as the world got more crowded, such as in what we now call sub-Saharan Africa and North America. The point is that a penchant for mistreating 'others' did not suddenly develop in human beings with the advent of farming, then civilization.

You keep referring to how people treated one another within groups (with which I agree, based on what I've read) to argue that 'other’ness did not exist. According to what I've read, it did exist between groups. It did not manifest itself uniformly, but it did exist. (One reason I’m looking forward to reading that book I referenced is that I have an idea that the peoples in this part of the world were working their way towards a uniform state of peaceful coexistence among groups when the Europeans showed up.)

More importantly, whatever the history of 'other'ness, given the fact of its existence today we do need a better approach to the governance of society. Replacing the ethical muddle that is old Liberalism with "A New Liberalism" featuring an ethic of justice that follows from the undeniable truth that human beings have no choice but to effect choices would be a good start. The structure of a Liberal society would be very much the same, but its functioning--how people got treated in it--would be transformed, especially in the economy.