Depends on what that person is doing, how loudly he/she is listening at and how in(tolerant) of noise the surrounding is. Obviously nobody should be doing this while driving or doing something else that requires serious attention. Llistening too loudly in the library is also socially disruptive. Obviously too, one shouldn't be wearing headphones/earbuds while attending an important meeting.
Other than that, public sighting of headphones is pretty much a normal thing for me.
I almost never leave the house without my portaPros. A freind of mine loves his HD202s so much that he wears them around his neck, often forgets he has them on, and many times hangs out with me, walkin' around Boston or something, with his headphones but without his Zen Micro, because he forgot it. It's pretty funny.
I voted yes, and in a city, it probably isn't a problem. However, in situations where you're likely to see people you know, I think it's a little tacky to go about with headphones in. I always get slightly annoyed when I see people on my small college's (850 students) campus with headphones.
If they don't want to talk to people, they can just ignore them - like I do.