In that part of this country in the 1800's racism was more directed towards the indigenous population. It would have been easy in that time and place for Norwegians arriving here from 'civilized' Europe to fall into that cultural trap. Once in that ttrap, racism corrupts culture in so many ways it can be hard to see all of them.

If you aren't familiar with Andreas Hofer, who posts frequently here in Medium, I highly recommend him to you. He has developed an explanatory paradigm of culture using foraging, herding, and farming as the primary formative influences, which he hypothesizes have gotten as ingrained as DNA (if not to say ingrained as DNA). According to him, (descendents of) herders and farmers woud be, for different reasons, more prone to racism than foragers would be.