Surely there were groups who were neither farmers nor herders who learned they could live by taking from others, led by what we would call today warlords. It is quite possible that warlords began 'protection rackets': food (and women?) in exchange for protection from all warlords, including themselves. Eventually, they could have imposed themselves permanently on farming communities and enslaved the farmers, then started building walls to protect from other warlords what they had taken.

That would be the start civilization--living in cities. That is the legacy of civilization to this day: except for those miniscule times (in the scale of things) when some notion of justice has prevailed, the history of civilization is 'rule by the most ruthless'. Defending justice against those who are the most ruthless is the challenge we are facing--or, to be more accurate, not facing.