I agree, I really don't like when they control me. It is sometimes hard for infjs to work with or have a relationship with estjs because I feel like estjs play mind games and infjs can easily detect mind games what the person is trying to achieve. It also bothers me when estjs are very critical which they can get to be. I like estjs though, it's just hard for me to get to know them and work smoothly with them.First off I don't think it's necessarily a negative thing to admit that you "don't naturally mix well" with another type. Although I know and like people from all different types, there are some that I have to work more than others to like, and that's natural. What I have found is that around certain people I just can't ever discuss certain topics or themes. Recognizing this beforehand paves the way for a smoother relationship in the long run.
In my life I have had a lot of trouble with ESTJs (though I have a lovely ESTJ supervisor who is very fair, supportive, and fun to be around). I got stabbed in the back by an ESTJ friend who took a random disliking to me towards the end of high school and did whatever she could to ruin my reputation. I have an ESTJ coworker who though I barely know her gives me the snake eye every time I try and smile and say hello. I constantly hear her bossing people around and being pushy, and I wonder if she doesn't like me because she can tell I can't be controlled (ie, may be as stubborn as she is). And my ESTJ sister-in-law who is very kind is also extremely strict and rigid (especially about her kids, who if you ask her are the best-brought-up and most perfect children in all the land, and she doesn't understand why everyone else's children are so poorly behaved, and you ought to be spending every second of your spare time with them when you're home because that's what's FAIR; it's only FAIR for you to spend 90% of your two weeks a year at home on playdates with her kids in spite of 3 other nieces and nephews and aunts and uncles and grandparents who are patiently waiting to see you, never mind the two hours of sleep you've gotten from traveling to be there and late-night xmas shopping...).
Sorry this got a little passive-aggressive and bitter.
I think the problem stems from their trying to control me. ESFJs do this too and it drives me crazy, but at least they sincerely believe they have your best intentions at heart. ESTJs just have this personal system that they believe is best and, still worse, truly believe is completely objective, and because they're extroverts they try to force it on you, too. I have major problems with anyone who does not respect differences and expects everyone to be like them. It bothers me more in ESTJs than in ENTJs because at least ENTJs as Intuitives think a similar way as me, so though I find them overly-forceful and controlling as well, they at least have a solid, Intuitive logic to back up their argument. The bad ESTJs I've come across rely on nothing but their own skewed (to my mind) ethics system that they believe is "objective" and think they have some God-given right to foist upon you. If they kept it to themselves like ISTJs I wouldn't mind it at all, but every problem I've had with an ESTJ is when I felt like they started disliking or judging me for some undefinable reason over which I had no control, and which was probably related to me not fitting into some mold they think everyone ought to fit into.
So, serious problems with ESTJs. I have learned certain things about dealing with them: a) don't debate about politics (I generally find them conservative, and I am very liberal). b) Try not to debate about anything. Avoid debates overall, or anything which will draw out their aggressive, competitive instincts, unless you enjoy dealing with that sort of thing. c) Keep your opinions to yourself; smile and nod in the case of an unavoidable debate. d) I hate being fake-nice and "keeping up appearances". But they respond well, almost seem to require, fake-nice. So do it if you must. e) Stand up to them calmly and rationally when a line has absolutely been crossed. They respond well to logic and reasoning, though they may not agree. If you become emotional they will write off your argument. I have observed ESTJs writing off their own emotions as well. f) Keep a certain distance from them, whatever is comfortable and works best, and only open up the most personal parts of you if you absolutely know and trust them. They will judge you, but how much you let their judging get to you is up to you. In the case of a good friend you can laugh it off and not take it seriously; in the case of an acquaintance, coworker, or relative it can make you resentful or self-doubting.