Audio as a hobby is the ultimate example of the law of diminishing returns. There is so much good equipment out there at a reasonable price, that when you spend substantially more, you're really getting only minimal improvements. Don't buy all the "night and day" annecdotal reviews, they're simply examples of confirmation bias at work. Yeah, a $2000 pair of headphones may sound better than that $300 pair, but undoubtedly NOT 700% better. The question becomes what can you afford and what are you willing to spend to eke out the last vestige of sound quality.
Aye. I also think, as Mal pointed out - you're probably remembering things better than they were (the mind is great at building things up like that). Also, I'll add that usually the first time you hear good material or a good source, there is a degree of "wow" that you probably won't re-capture (it doesn't mean the quality in the future is gone, you just sort of habituate to it).
On another note, my favorite pitfall is when someone wants to spend $1000 on a setup, seeking great quality, but they're playing the same music they were playing from their cell phone, from iTunes, or YouTube rips, or some variable rate MP3's they downloaded off some torrent site and then go and say that the equipment "isn't that good."
You have stumbled a phenomenon that many of us have. Amping in general is over-rated. That is not to say it does nothing but it is WILDLY exaggerated here. It means people spend hundreds of dollars on amps and run them out of cheap DACs and expect miracles. It's to be expected that a $100 DAC doesn't sound as good as a $10,000 CD player.
+1. There's a lot of wildly variable reports on what can and cannot be accomplished with amping - I agree with armaegis; keep things reasonable, and enjoy the music. Once you find whatever headphone you happen to jive with, then find whatever amplifier suits it, and the rest is history.