YOU HAVE OVERCHARGED YOUR EXTROVERTED BATTERY LIFE. Just kidding, you will probably realize the need for alone time after extreme social exposure or consistent penetration of external sources. It's your inner INTP speaking...
You don't like being around others when not confident because you gauge yourself through other people, not within yourself. You take things in externally (ESFJ values are primarily gathered externally). If you are not happy with yourself, you no longer wish to be around people, as they are usually a very important part of your external living. ESFJs also tend to see themselves through the eyes of other people, whether or not the things in question are actually seen by those people. Unconsciously, I think about myself through other peoples' eyes, which makes me very flexible, always changing, and in some ways lacking consistency. When I feel self-conscious (whatever it may be), I’m certain other people will see it clearly and it makes me uncomfortable. Lately I have come to realize that these perceptions are mostly only visible to myself, and the people I'm around don't always share the same opinion. In other words I’m too hard on myself. It's our damn J function. Always changing how we feel and then feeling it too solidly and strongly. It has to be decided, always strictly one way, and that doesn't leave much room for contemplation. Once we think it, we're set on it, and after several bouts of changing what we think as we gather new information, our confidence can become deflated.
It's kind of a stabilizing mechanism in a way, to make sure that nothing is in the dark; we think the worst, making us internally pessimistic. But it's a different kind of pessimism, a way for us to establish order and protect ourselves from the worst. By thinking the worst, and setting ourselves to higher standards, we aim to avoid bad situations and unwanted traits/habits. And in this way we can also lose sight of the positive traits we have. Then our confidence goes out the window yet again.
Sorry for writing way too much/over-explaining.
And this may not even have been what you were talking about. But if it was, I hope it helped.