There is no word for a male mistress. Mistress is defined as an adulterous, kept woman, "a woman having an extramarital sexual relationship, esp. with a married man" (Google).
Boy toy is not adulterous. And doesn't convey at all the same position in someone's life as mistress.
Adulterer is a more of a legal term.