Do "I" really need a headphone amp?

Yes an external headphone amp will improve the sound quality. Amps aren't just about volume. They're also about control. The HD650 is quite amp dependent. I think a neutral solid state amp like the Gilmore Lite or a dynalo amp does very well. Especially with bass control.

The RME is great as a sound card, but very bad as hps amp ( I had the RME Digi 96/8). Even in baby face hps amp is on the face. 

Can you define "not too expensive?"

 

With a FiiO E9 you can get gobs of power for $112.
From my experience more expensive amps don't really add anything at all. The most important thing is that the amp has enough power, and the FiiO has plenty.

You can get something more fancy, but I doubt it will improve sound quality much further.

 

I hope this will explain in great details what I meant.
 

Soundcards typically have highish output impedance (the RME is 75 ohm) and little/no amplification headroom (e.g., no room for meaningful caps on a card).  Thus, they do not do well as headphone amps.

I have the HD650 too, some people use them on sound cards & ipods, but I have found like many they sound best when you give them a proper amping, but that costs money.

First you'll need to decide if you want go with tubes, or solid state amp.

For a little more you can get (here it comes - I sound like such a fan boy) the TTJV Slim. Of all the audio components I've purchased, I am extremely happy with this one. See this current thread: http://www.head-fi.org/t/577284/ttjv-slim.  I think it's better to spend more on something you'll really be happy with and keep, than to settle for less, be unhappy, and end up spending more anyway. But - if you're not really going to carry around your rig, you can get more bang for the buck with desktop gear, an avenue I am seriously considering.
 

 

Volume is not the only thing. For one thing, output impedance is really important, and as was mentioned before soundcards typically have a high output impedance. This affects the frequency response of the system, because the output impedance will act as a voltage divider. This is not always bad, but headphones such as the HD 650 have different impedances with different frequencies, and hence the sound is altered.
Then there is also distortion. Amps that have more power can drive the same volumes with less distortion, this is especially noticeable if you drive an HD 650 out of a low voltage source such as an iPod. (usually around 1.3-1.6V). It won't sound as good as on a proper desktop amplifier.

The specs of the E9 are excellent, and will drive any headphone that is not orthodynamic or electrostatic with ease. It is essentially a fairly transparent amplifier that just amplifies the signal without altering it much.

There are also amplifiers that have a specific sound signature, and alter the signal in addition to amplifying it. This is especially true for tube amplifiers. Some people like this, and this is what from my experience is usually experienced as a better sounding amplifier. Personally I think the best way to alter sound is through DSP's (sound effects) and equalizing or through the headphones themselves. e.g. changing headphones makes a dramatic difference in sound compared to spending the same amount of money on an amplifier.
I personally have trouble telling my four headphone amplifiers apart. That is, a vintage stereo receiver, a portable amplifier (AMB Mini3), a discrete solid state amplifier (Audio-GD NFB-12) and a tube amplifier (LittleDot MKIV). You can blame it on my ears or skepticism, but I truly believe that amplifiers only make a marginal difference.

Once they are good enough, there is not need to upgrade. It is very well possible that your soundcard is not good enough. The FiiO E9 will however be without doubt good enough for your needs. You can always try upgrading later, but when you do make sure that you can return the product at a minimal restocking fee if you can't detect a difference.

Just my opinion based on experience.

Also: amplifiers work so fast that latency is never audible. Connecting it is as easy as connecting a wire. If the soundcard has RCA outputs then you're completely fine. If it only has TRS jacks, i.e. 3.5mm (1/7.2") or 6.3mm (1/4"), then you might need to buy a RCA to TRS cable. That will set you back roughly $5-10.
Edited by Tilpo - 10/26/11 at 9:52am

Yes, connect the analog outs of your card to the E9 and plug your headphone into the E9.

 

The amp will provide a substantial upgrade.  In the unlikely event you do not find a difference, simply return it.