There are no audiophile-quality portable players that I've heard so far. Not the iPod, not the X5, and not the Zen Micro, each of which I've owned at one point or another. The closest I've seen to audiophile-quality sound in a portable were some vintage Sony PCDP's, but that's about it.
As a current owner of the X5, I can make a few comments:
It has a pretty even sound, with good treble and bass extension, slightly bright. Clarity is about average, but it does manage to convey a good soundstage with music that actually has it. Overall, it's fairly similar to the Micro.
However,
The quality of the player is offest by it's ability to play lossless. I don't know about people not being able to tell the difference with 320k mp3 and lossless - I know I can't tell the difference with music that I either don't know or am only marginally familiar with, but with music that I know by heart, on gear whose sound signature I am familiar with, I can tell the difference. I've done DBT with my friend and I've been able to nail the 320k file and the lossless file every time - with music that I know well. While audible compression artifacts are kept to a minimum with 320k, there are differences in soundstage, detail, and texture that give 320k away.
Based on that, I would recommend the X5 over the Zen Micro, simply because using .wav files on a 5 or 6 gb player is not very practical. I don't have experience with ALAC, so I won't say whether or not the iPod with ALAC is better than the X5 with FLAC.
In any case, if you're looking for genuine hi-fi playback out of a portable system - forget it.
IMO.