The short vertical parts adjacent to it also reach into the magnetic field and should experience forces. why can we neglect them?

It is not only the horizontal part of the loop that dips into the magnetic field. We can neglect or disregard the horizontal parts of the loop that hollows into the magnetic fields since only the parts perpendicular to the magnetic field complement to it.