Good thread Victor. I've been pondering many of the things discussed here for a while now. What brought it to a head for me was wanting to have a good-sounding travel rig. As others have pointed out though, it isn't as straight-forward or absolute as amps making a significant difference or not, as it depends much on the headphones and source.
On the one hand, I have the new Sony XBA-3s. While I do have a Pico Slim (I had, way back, intended to get customs and might still in the future), I don't find it benefits the Sonys in the slightest, but in fact, adds a bit of unnecessary bass to the sound, without any refinement in any other area. So I only use them directly from my iPhone.
On the other hand, I own RE-ZEROs, which sound terrible out of my iPhone and far more pleasant via the Pico Slim. This is, from my understanding, due to their impedance and sensitivity.
I ended up taking a chance on a Fostex HP-P1 so I could bring full-sized headphones with me when I travel. The result was good -- better than directly from my iPhone for the headphones I was taking with me. However, at the last minute before leaving, I had the chance to try the new Sennheisers, so I picked up a Triad Audio L3 second-hand. Direct from an iPod, it is a lousy amp. However, when I used the line out from the Fostex, the result I feel is fantastic -- clearly a better amp than what is built into the Fostex. However, now I have a rig that is seriously expensive (though no $200 LODs!).
I also had the chance to try a Continental direct from an iPod, and liked the results with that. I wonder if it has to do with the different input impedance of each amp.
Was it worth it? On the one hand, I think it's great that I can get something approaching the performance of my home rig, or at least very good sounding, especially with many new great portable headphones available around the $300 mark. I wonder how much of that has to do with the Fostex being able to get a digital signal from my iPod or iPhone, especially because my money is going into the electronics and design, and not expensive LODs and capacitors. Maybe I just struck it lucky with my choices? I do wonder how much bang-for-the-buck one could get with AC-power only sources and amps for the same money. What am I paying for in my (trans)portable rig? Miniaturization? The batteries?
There are many questions I want to be able to answer now.