Personally, the only thing I don't like about wearing headphones outside is that it puts an awkward dent in my hair. It doesn't really work unless you wear short hairstyles.
IMO, portable headphones are fine in fashion, but not full-sized cans. Good fashion is always based the idea of portraying an image. If you wear headphones out, the image you want to portray is that you care about music and are knowledgeable about sound quality.
If you wear open headphones outside, you look bad because the sound won't be good in noisy environments, so it makes you look ignorant or just dumb. If you wear huge full-size headphones, you look bad because they're impractical and generally need serious full-size amps to sound good (even if they don't need one, they LOOK like they need one, which is what matters in fashion), so you'll look like you're dumb or don't know how to spend money.
The rest of your clothes also affect how the headphones will look. I.E., if you wear a preppy or clean-cut getup and add headphones, you give off a nerdy/uptight vibe that doesn't really look good. On the other hand, if you add headphones to a laid back "music-inspired" look, adding headphones around your neck or on your head looks good.
For comparison, The classic "ipod" look is: colorful tshirt, loose khaki shorts, flip-flops/sandals, mp3 player in pocket and earbuds with white cord, and add whatever accessories you regularly wear. This look is really easy to pull off which is why you see so many people do it. It portrays an easygoing, laidback, "fun" personality.
If you wear headphones instead of earbuds, you can't do the laidback "ipod" look because you will draw attention to the headphones, so you need to build a look around that fact. You can either go for the "music lover" look (band shirt, slim jeans, canvas shoes, etc.) or if you are up for a challenge, you can try to portray the "I'm a DJ at night" image (much harder to pull off): urban-styled graphic tshirt, optional white undershirt, wrist-watch with neutral colors (silver/black/darkbrown, some red or white on face is okay), dark fitted jeans (no skinny jeans), sleek sneakers with colors that match or complement tshirt design (red works well these days)