Perhaps the most anti-slavery president was John Quincy Adams. Andrew Jackson accused Adams of stealing the presidential election due to a corrupt deal with Henry Clay. Jackson was a slave owner. Abolitionists voted for Adams.
To this day many Native American Indians will not accept a $20 bill, that has Andrew Jackson on it, due to the Trail of Tears, that they blame on Jackson. Communications were poor back then, so in Jackson's defense maybe he didn't know what was happening. Some Indian tribes even fought with Jackson during the War of 1812. Some tribes fought with the Confederacy in the Civil War because they viewed the Union and federal government as an extension of Andrew Jackson's legacy.
Jackson did stop a secession or nullification movement in South Carolina. But otherwise he was pro state rights on the issue of a national bank. Jackson preferred state or local banks. Harry Truman seemed to like Jackson for that reason, as Truman hated banks.
General Andrew Jackson was viewed as the winner of the War of 1812, in many ways like George Washington was in the Revolutionary War. Successful generals were popular as presidential candidates. Every president from after Andrew Johnson until Grover Cleveland was a general in the Civil War.