I've been using the PMD661 for about a year and it's certainly a versatile equipment.
The recording capabilities are stellar: 2 xlr mic preamps with 48v phantom power, S/PDIF, and 1/8" line in.
As far as using this device as a portable source - it can definitely work. Although there is a small hiss in the headphone amp, once the music begins it is not noticeable, and the amp is powerful enough to drive headphones that require more power, such as the Sennheiser HD-650.
The player supports formats up to 24/96 and the details you can capture on this device is quite amazing.
The only drawback for making this your source is that you need to get the music onto the SD card, either recording the tracks on the device in real time or through your computer with things in wav format. Just an additional step, especially for those that are using iTunes, and can be considered a hassle for some.
Obviously with so many capabilities, this device may not be practical for the general public, but for those of us that work with sound on a daily basis, this is an incredible sounding device that can deliver the quality that you're striving for.