What percentage of Northerners do you believe had a strong interest in ending Southern slavery? I'd guess that not many were so concerned. Like most soldiers, Northern soldiers were likely doing a job and/or fulfilling a legal requirement through their service — in order to advance their own personal interests.
Second, just because you think slaves should be free does not mean that you want to compete with them for jobs. Racism thus becomes a way to excluded competitors vying for the same job as you. Moreover, the business interests which controlled those jobs likely had an interest in fostering continuing racism: to maintain a permanent underclass capable of replacing white workers; with such a threat looming, white workers would be less likely to demand higher wages, etc.