Something to bear in mind with regard to audio is that cost is not always in step with sound quality. As you spend more you can expect nicely machined enclosures, more complex circuit topologies, and state of the art parts. However, more often than not, the differences with amps is more incremental than dramatic and not always related to cost. With todays digital compressed sources, the most revealing equipment can actual work against you by making the limitations of those compressed files all too audible. This can be a pro or a con depending on what you like to get out of the audio experience. I'm not necessarily saying that a pricier amp won't be better than a less expensive, simpler design, but beyond a certain point, those improvements come at an increasingly higher cost. Sonic imporvements are governed by the laws of diminishing returns which means that beyond a certain point, incremental improvements often come at significant increases in cost.
Fortunately there are some sonically excellent offerings on the market, such as the Pocket Amp2 and Go-Vibe. For a first amp, I think they would be the best option. They may be all you ever need. If you do upgrade, you will at least be ready to appreciate why the upgrade is better. However, don't assume a $400 amp is 4 times better than a $100 amp. It doesn't work that way. On the other hand, a quality $70 amp is worth way more than listening unamped with most other portable gear.