Why would surface waves be more damaging to buildings than P waves or S waves?

Because the amplitude of surface waves diminishes less rapidly with distance than the amplitude of P or S waves, surface waves are often the most important component of ground shaking far from the earthquake source, thus can be the most destructive.