I just listen to music for the enjoyment of it and try not to punish myself if I'm not listening as intently at times as I do at other times. Sometimes your mind wanders to other things and that's Ok too.
I really don't want my hobby to ever become a job. Even when I'm listening to music with a new pair of headphones or if I've switched in a new source or amp, I try not to make a big dal about it. If someone asks me how I like my new Headphone X, my answer is often that I really don't know yet.
Until I've had 10 to 20 hours of listening with different types of associated equipment and different types of music, at best I'll just have a basic gut reaction and won't be able to say much else about Headphone X. Otherwise, I'd end up driving myself nuts trying to form crystal clear opinions about the gear instead of just enjoying the music on whatever gear.
I think the same logic extends to my listening "skills" themselves. There is no real reason to attempt to "train" you ears as such. Maybe it would be fun to take a couple of the available tests to see how well you do, but why then punish yourself over the results? Or then try to push yourself to get better? For what?
I used to love playing golf until I get too competitive within myself and I ended up just giving up altogether. Same would be true if ever I tried to perfect my driving skills when on the race track with the heel/toe technique and the like. It's more fun just to practice basic lapping skills for me, or to go out a play a round of golf without even keeping score just for the fun of it.
So I'm sort of that way when it comes to pretty much anything that is a hobby. If I make it work, I end up enjoying it much less, and so it is with audio and listening skills. Better to let them develop on their own as your foot is tapping than to try to "train" your ears and then suddenly you can't just sit down and listen to a piece of music without analyzing it to death.
Not to say that you won't enjoy doing this, but on those rare occasions when I've tried, I've always found it to be rather tedious and tiring. Don't say you haven't been warned!