Listening to music isn't a waste of time. However you choose to fulfill your life, as long as it is something you enjoy, isn't that what life's all about?
I have many hobbies: dirtbike riding, computers, video games, watching/playing sports, listening to music, etc. Sometimes I can combine two of them, like listening to music while I tinker with my computer. Many times music helps me get through some of the stuff I need to do, such as work, homework, long drives in my car, etc. Other times I just love to sit, relax, and listen. Sometimes I even listen when I sleep.
Nobody has the right to tell me that I'm wasting my time listening to music. At least, no more than I have the right to tell them that they're wasting their time with their hobbies. If I feel that what I am doing is time well spent, is something that brings fulfillment and enjoyment into my life and makes me happy, then that time absolutely not wasted. If anybody tells you that you're wasting your time listening to music, politely suggest to them that they waste time when they play video games, or watch TV, or whatever it is they do for fun and see how they react. It's all subjective, and it's different for every person.
I can see where some people could possibly have some objection to it. As with any obsession if you slack on personal hygiene, work, exercise, etc., in order to spend more time with music, I can maybe see where they might have a bit of a point in that you might be taking it to an unhealthy level. But still, in the end, it's really up to you.