In most patricarchal societies the vast majority of males are destined to being outcast, or locked in a brutal fight for survival through dominance. It's pretty much the norm in the animal kingdom. The silverback is better off than most of the females, sure. But you tend to forget all the male Lion Cubs that get ostracized or left out because of their perceived threat.
There has never been much room in society for beta males. Animal or human.