To me the headphones are the most important part of the equation, then source and amp. The conversion for electrical to mechanical or back, like with microphone, phono cartridge, or headphone/speaker is the hardest part to get right. The HD598 are okay but not the most resolving headphones out there, and so improvements in source and amp will make a smaller difference than if you were using HD600, LCD-2, HE-500 or HD800.
But even then, the differences between a $100 amp and $200 amp won't be much greater than improving that to a $500 amp. On the other hand, comparing the bottom rung amp to the top of the line amp will be more striking than comparing incremental improvements, but we're talking about much bigger price jumps. The nice thing about the EF2 amp is that with mid-fi and even hi-fi headphones the amp won't let you down, much, vs more costly ones.
Again, I think the EF2 amp is better than it's DAC, and that could be the weak link rather than the phones if your PC has a good sound card.