Why was the Indian removal act considered controversial?

The Indian Removal Act took place in the first half of the 1800s under the presidencies of Andrew Jackson and Martin van Buren and directly led to the Trail of Tears, which resulted in the death of many members of tribes in the American South. It was considered controversial because the Supreme Court ruled in favor of the tribes' cause wherein the state of Georgia (which was actively seeking to evict the Indian inhabitants) was told it had no right to force their removal. Nevertheless, the president ordered their removal.