I do hate it, but I think it's less about the more people thing and more about the manners thing. Common courtesy is to ask before you invite someone else, so when people bring more people without asking, they're basically saying "I don't care whether you have planned for a certain amount of people." Rude as hell.
However on occasion I do get annoyed with one of my friends, because we'll plan some hang out time (we don't see each other often) JUST to catch up, and then she'll call last minute and invite a friend of hers, who I don't mind...BUT when this friend comes along, the two get drunk and I don't get to catch up with my friend. Last time she tried it I just said we could hang another time.