Lots of people I know don't like what I listen to.. but I've come to terms with the fact that I listen to some whacked out stuff that the average music listener probably will not be able to get into without some serious and genuine desire.
My brother is totally indifferent to music. He has told me straight to my face multiple times that it's just background noise for other activities. However, I don't really blame him.. he has an extremely busy life right now and actually used to be really into music back before he got married, had a kid, bought a house, bought a car, got a demanding job, etc.
My mother takes interest in what I listen to.. I mean REALLY takes interest - she wants to listen to everything I have. Some stuff she likes, some she doesn't, but she manages to appreciate all of it and has started to see modern music under a new light. I'm lucky to have her.
With the exception of one other person, the remainder of my family has a passing interest in music.. and basically none in my music.
My friends back home in Northern VA share some of my musical interests, but 99% of my friends here in CA just hate it. The only person here that even attempts to appreciate what I listen to is my roommate's wife.. and that's cool.. even one person is enough for me. I'm always burning CD's for her to listen to, and she always comes around with welcome feedback. She normally listens to stuff like Broken Social Scene and the Brazillian Girls, but I had her listening to Dark Tranquility and Plaid. I really appreciate people like that.. since they're so few and far between.
Nobody at work has any interest in what I listen to either, except for one guy who made a positive comment while I was listening to Bubblemath one day. I was glad to hear it.
Personally, I couldn't really imagine not having an interest in music.. but then again, 10 years ago I couldn't have imagined myself listening to what I listen to now.