Usual reasons given are:
Jackson was a slave owner
the Trail of Tears: Jackson was president when Native American Indians were forced to move West, and many died during the relocation. Some Indians today refuse to accept a $20 bill in any financial transaction.
In today's PC world, Jackson would be considered a racist white supremacist.
John Quincy Adams, probably our most anti-slavery president considered Jackson unfit to be president.
Other people seem far more worthy of being on our currency, such as Harriet Tubman, abolitionist and major figure in the underground railroad, helping runaway slaves escape from the South to the North. She is scheduled to replace Jackson on the $20 bill.
In Jackson's defense, he adopted an Indian boy, and some Indian tribes fought with him at the Battle of New Orleans which won the War of 1812, sometimes considered to be our 2nd war for independence. Given poor communications, perhaps Jackson was not aware of the fatal consequences that occurred during the Trail of Tears. While a slave owner, Jackson was little different from most of our Founding Fathers, other presidents and much of the world's leaders at that time. Projecting our current ethics backwards onto a different era may not fairly judge people of that time period. Harry Truman admired Jackson as a strong leader and president representing the "common man." Jackson supported the federal government against South Carolina's attempt at nullification.