Although the atrocities against African Americans were well known, why was so little done to prevent them?

Maybe people were scared to act out and stand up to the majority? We have seen many atrocities against not just African Americans, but look at what Hitler did to the Jews. How did we let that go as long as it did? We need to stop being scared of what others think and speak up against ANY atrocity happening to ANY race!