How was a successor to Muhammad chosen?

First: a successor for Muhammad had to be chosen, because none of his sons survived to adulthood, so a hereditary succession was not in question.

A successor to Muhammad (or generally a religious and political leader) is called caliph. The first caliph was chosen by recommendation from many of Muhammad comrades, most notably by Umar (who was a caliph later), and it was Abu Bakr.

The choice was disputed and some believed it should be Ali should be the rightful successor.

In general, the successor was chosen by recommendation from others and choice among the leading figures.