Depends. I'm a fairly avid gamer but until recently I'd been spending most of my time listening to music when I would have been gaming. Over the Christmas period I've found myself gaming again, but listening to less music as a result.
I guess they're both a waste of time in a way, but by that definition so is visiting art galleries, going to the cinema, reading books etc.
I do bouth whne on PC and use me headphones when playing games, I find it evens out.