Okay to plug HP am directly into component (non portable) full fledged CD player like a Rotel?...

It depends on the source. 

Most headphones take only a few mW to run to freakish levels - even high impedance headphones. Its not a problem of power, but that the output stages do bad things when they see a low impedance load. 

Some companies pride themselves on having sources that can drive headphones directly. I cant remember who at the moment, but want to say Krell or Wadia. Professional studio gear is generally able to drive 600ohm headphones directly (if they follow the studio standard of putting OMFGvolts into 600 ohms they better be) - provided there is a volume control somewhere in there. 

With the addition of an output transformer designed for the load & source the world is your oyster. 

  So were you asking about plugging your headphones directly into the cd player or do you mean, like I thought you meant, plugging the line level output of your cd player into an headphone amp instead of going through the receiver and then the head amp?  

I took it as the latter... plugging the headphone amp directly into the CD player instead of using a tape out or something on a receiver or preamp.

Plugging it into the CD player directly will eliminate one set of interconnects, the jacks, and any active circuitry in the receiver/preamp that needs to be on in order for it to output to the tape out. Most receivers or preamps will have to be turned on in order to route the signal from the CD player to the tape outs. Some, like my previous Counterpoint preamp, did not need to be on... the signal was going from the input jacks thru the selector switch and directly out.

Downside to going direct from CDP to HPA is lack of input switching, and inconvenience of having to move cables, etc. if you want to use the CDP with the receiver/preamp.