I'm not sure of your budget, but have you considered buying a quality integrated amp either new or second hand? I can highly recommend the Creek Evolution integrated amp to you. It is a very well regarded piece of equipment. I researched the hell out of things and auditioned things before buying the Evo. I was moving from a really well respected NAD system that had the classic 2600A power amp with I think it was the 1400 pre-amp which was really very good. I had a brand new NAD CD player, forget the model, near the top at the time (2007).
The Creek doesn't even have tone controls, but man does it have a very nice, balanced sound signature. As for headphones, the headphone section runs right from the main amplification section with a large resistor in between so you don't get an "afterthought" headphone section. I have a really nice pair of Energy C5 speakers paired with my Evo and when using headphones I have some Grado HF2s.
Anyway, I have just ordered a headphone amp, the Valhalla, not becuase I need one, I just wanted to have a new sound signature. A great integrated amp should always be seriously considered when on a budget and the Evo provides you with 85 quality watts of British engineered sound. You can find the first generation of Evolutions used as people get upgrade itis, I know I have it, and if you can get it for a good price then your golden. It doesn't come with the phono section, but one can be added so no worries.
The audio shop I purchased mine from planetofsoundonline.com has a good article from the owner where he writes about the Evolution. Before buying it I thought he was perhaps being a little biased, but after 3 years with this little gem I have no regrets at all. Just something to consider. Good luck in your search.