Which factor most directly affects potential energy?

Mass, gravitational force and height

Explanation:

Potential energy of body is the energy at rest in a body and it depends on the mass, acceleration due to gravity and height of a body;

Potential energy = mass x gravity x height = mgh

The more the mass, gravity and height of a body, the higher the potential energy a body will posses. This is why a stone at a height has more potential energy than one on the ground.