In toe early stages of the war, African American men were not allowed to join the Union forces or the Confederate forces. Why?

At the time African Americans were seen as slave, as well as people without rights, because of their skin color. One of the main causes was a 1792 law that prevented them from bearing arms. That mean't for military purposes as well. Once they by passed the law only free black men were allowed to join.