None of my friends are INFPs and really I don't think I've ever met any INFPs in person so I've never had this happen to me, but I can definitely understand the frustration. I would love to meet an INFP and have a friend I felt I could really connect with, but I don't really fit those typical INFP interests either and I could see it being a roadblock. I only read when I'm really interested in a topic and I want more information than I can find online or occasionally when I hear about a novel with a really interesting story, so maybe like 3-4 books a year max. I rarely play video games and when I do it's usually older games that I grew up playing. I know absolutely nothing about anime and I can't remember the last time I drew something.
My pasions tend to lie in things that can be experienced or knowledge that can be applied in the real world too. I'd love to travel someday or even just go out and experience more of the things my area has to offer, but I'll have to do it alone because my boring friends never want to do anything but hang out at each other's houses. (I actually have the opposite problem to yours - my friends aren't too fantasy-based, they're too reality-based. They're not really interested in much beyond what happens in their daily lives and it gets boring fast.) I love music and going to concerts (sometimes to dance to but mostly the kind that I can really connect to) and comedy (especially stand up). I also really enjoy researching and learning new things particularly things related to science, psychology, or culture.
Part of me thinks that just knowing somebody who has the same level of passion as you can be very rewarding even if your interests are different. I love to learn so if I met somebody who was very passionate about an area I didn't know much about it might be interesting to have them tell me about it (especially if their passion for the subject makes me feel more connected to the topic - just like an enthusiastic teacher can make a boring class fun). On the other hand I can see how feeling so passionate about something yourself and not having anyone to share that with can suck. I know there are times when I hear a great band or see a great movie or read an article about somewhere I'd like to go and think Wow, that was great. I just wish there were someone I could talk to about it.