I have to disagree here. It depends what you want to get out of listening to music -- if you really want to expand your horizons in classical music, you might listen more carefully than if it is like background music with distractions. Yet you shouldn't approach it like a textbook.
However, I suppose you don't just listen to the music for ten hours per day. If this is the case and you're having trouble "understanding" the music, then take an hour or so to listen through a whole quartet or perhaps a whole CD without any distractions -- no book, no surfing the 'net, no chatting with others in the room. It took me a few listens to understand Bartok and I can now listen critically to live and recorded performances. It adds a lot of fun to the live music experience when you can sit and listen for the parts of a piece you particularly love and see how the group plays it. And if you like it more or less than your reference recording, what was different? Did the first violinist play too mechanically? Did the cellist recede too much into the background?
The process of adapting to new genres within classical music can be very frustrating for a beginner or someone who's listened his whole life (like me). I started listening to atonalist music earlier this year and bough some recordings of music composed by early 20th-century masters Alban Berg, Arnold Schoenberg, and Anton Webern. My first acquaintance with this kind of music was at Boston's Symphony Hall. The BSO played Schoenberg's
Five Orchestral Pieces and somehow it just struck a chord within me even though I couldn't make heads or tails of it. It seemed like random notes strewn about the hall without any order, but I still enjoyed it. This sort of thing is a good sign. Like with a difficult poem, understanding is not seminal at first reading but rather attachment to some element within the work that piques your interest or just makes it appealing to you.
With a musical work that you're trying to understand, I'd try the following -- try to identify voices within the music. This can be especially good for chamber music where you have only several distinct voices. You can even try to attach human moods and personalities to each voice. For example, a violin often sounds like a complaining woman with a soprano voice. A cello can sound like a lazy old man who is responding to his daughter's complaints with a feeble defense.
Once you can assign character to the music it becomes much easier to understand. It gains meaning for you and each time you listen you can try to find the characters you created, only to find that you hear a new character in the place of the one you heard the last time you listened. Or perhaps you hear a different quality in the "voice."
Sorry for rambling. My point is that you can take it easy if you want, but there are ways to look deeper into music so that you understand it better and can move onto more complex and progressive styles. Looking for voices is, of course, a much more difficult exercise in later 20th-century works and sometimes quite futile. Try different tricks of your own to find meaning and music and see to how many different pieces you can apply them. Good luck!