Then you are in business Jay. I use my Touch with various amps with great results. Don't let the PA2V2 fool you. Gary builds some nice amps that last. I have had mine over 5 years and still going strong.
I would have to go with the XM5 for all the options it has. Stay tuned, a lot of my iems have not gotten some love recently and they are complaining, so I'm thinking about selling some of my portable amps. If you're interested in any, shoot me a PM before I put them up for sale.
The HFI 580's benefit from amping with more quantity bass(depends on the amp but generally) and other nice little responces like slightly more forward vocals and depending on the amp, the highs could get a bit less harsh by a tad bit than compared to using the HFI 580 with an iPod.
I prefer FiiO's with their bass boost and universality and musicality with the HFI 580 which isn't looking for accuracy. I would rather use the FiiO E17 as opposed to the O2 with the HFI 580's. The O2's were a bit too clean and didn't compliment the HFI 580's with it's EDM and Rap and electronic genre preferences that the E17 was musical to go along with. The E17 however wasn't as energized and sometimes loses out on speed and bump and just head shaking EDM when it can't "keep up" in a sense and that's where the desktop O2 or something could rectify. The best of both worlds for me is in my NFB 12.1. Musical, and energized, although some of that cleaness is gone, I can just line out to the O2 if I really ever wish.