Yeah it is disappointing. Hopefully a talented kernel developer will figure it out.
Here's my rig as it stands now:
Galaxy Nexus with 64GB external flash storage -> Pico Slim amp -> HIFIMAN RE-262 IEMs
Once you root it, you can use an OTG USB cable to connect a flash drive so you have tons of storage for lossless files (it's a shame the GNexus doesn't have a micro-SD slot). You can use an app like PowerAmp to play lossless files.
http://imgur.com/zHJIh
http://imgur.com/7EpbS
The easiest way to root your Galaxy Nexus is to use the Nexus Toolkit, a free Windows software you can download here: http://forum.xda-developers.com/showthread.php?t=1392310
Just install the toolkit, and follow the steps to root the phone. Just note that the rooting process requires you to unlock the bootloader, which will wipe your phone's apps and data. Fortunately, the Toolkit lets you perform a backup of your apps and data first, and then you can restore them after the rooting is complete.
If you get stuck, that XDA thread is a good place to post questions.
Once you're rooted, you can flash custom ROMs and custom kernels. Personally, I just keep the stock ROM because it does everything I need it to do. However, I did flash a custom kernel (Franco Kernel) that give me some audio enhancements. He has an app that downloads the kernel, keeps it up to date, and lets you alter the settings I'm speaking of (https://play.google.com/store/apps/details?id=com.franco.kernel&hl=en).