The people who finally drove the rulers of aksum from the coast of africa were

The answer is: Muslim Traders.

Aksum remained as a strong power until the rise of Islam at the seventh century. Alhough his relations with Muslim neighbors were good due that Aksum gave refuge to early Muslims, the rise of Islamic empire took control of the Red sea shores and most of the Nile for commercial purposes, putting the empire in a position of isolation