Which headphone and headphone amp do you use?
Also, do you have a 2 channel system that you could use the Prima in? This is where I've found the Prima to really shine the most, and I rarely use it with headphones anymore. If what you are saying is that $1,800 is a lot to pay for a CD player, I think that very few people would argue with you!!! But the $1,000 used Prima (if you can still get it) seems like a much better deal.
To decide how much you can (or want) to spend on your source, you would need to put an overall budget together for your system and then determine how much of that budget you are willing to pay for the source, headphones, headphone amp, and cables. Again, you will find all sorts of opinions on this, but I'm in the camp that believes since it all starts with the source, this is where you can get the most improvement in your system.