This isn't going to helpThere is an old adage -- A chain is only as strong as its weakest link.
That's not entirely true in audio. Sure it's true, but it needs to paraphrased just a smidgeon -- A chain is at best as strong as its weakest link.
In other words -- yes, if you have a crap source, and superior amplification, drivers (basically, everything downstream), then you're going to hear the weaknesses of your source, most likely.
But it's cumulative. So, for example, I have a really nice source, amp and speakers, but I'm using a crap pre-amp right now. So there's two possible weakest links -- the pre-amp, and the CD's I play.
It's funny -- the only reason the pre-amp is crap is because the amp amplifies so much that I can hear low-level noise (probably due to Brownian motion amongst the atoms and electrons). Well, if the dither is inordinately high on the CD, I can hear that, too.
Guess which is more annoying? Yup, you guessed it, the dither. Even if it's quieter than the random-EM noise eminating from pre-amp, it's still audible, and more annoying.
My point? I guess what I'm trying to say is that you need to add weighting factors to various things, as they are important to you. If you're perfectly happy with mp3 compression, then a MP3-based source might be perfectly adequate.
I've heard quite a few headphones -- even real cheap ones -- improve greatly just by throwing a really nice (read: completely overkill) amp at it.
So, short answer: it depends.