Generally the headphone will be most important hands down. But it really depends on the efficiency of the headphone as well. If the headphone is something that's meant for mobile use and doesnt require a lot of juice to power it, then by all means u should go for the headphone first.
On the other hand, if you want a headphone like the sennheiser HD800 which needs a ton of juice, using something like an ipod on it is gunna make it sound like a broken 20 dollar headphone.
So generally speaking headphones first. But if youre going more hi fi, maybe player first.
What kind of headphone were you considering?