Why did Madison believe that the spirit of party and faction would have a negative impact on the government?

One party may feel like the government is "disregarding the rights of another or the good of the whole" (Madison). Therefore creating conflict between the two parties; also creating conflict within the country.