Who suffered the most due to the lack of unity in the Federal government during the years of Reconstruction?

Those who suffered the most were notherners and freedmen and freedwomen. They suffered most because after the war during reconstruction everyone started focusing only on south because of all the wealth that it could bring and nobody cared anymore about the people from the north, what was important was to remove traces of confederacy from the south.