Either maintaining the electoral college system or eliminating it must necessarily privilege one side or the other in our dichotomous political sphere. Neither position can be justified morally without presupposing the moral superiority of one side or the other.
The best we can do is majority rule with due respect for the rights of the minority. Doing away with the electoral college would not divest anyone of a single right.
It must be noted that, whereas the perpetuation of slavery was a result of the electoral college system, the rationale for it was actually anti-capitalism. The idea was to prevent the more populous, manufacturing-and-commerce-oriented states from running roughshod over the less populous, agrarian states, whether they allowed slaves or not. That continues today with the rural-urban divide in our politics that roughly coincides with the conservative-liberal divide.