[bagwell359]

Thanks for taking the time to check! I understand this thing's primarily a speaker amp

Not so.

and maybe the fact that the headphone out is just the speaker power resisted down causes some noise?

The difference in output is due to the load of the headphones in balanced. SE is another story, because its not balanced.

I plan on using it with speakers but I use headphones probably 70% of the time so I was hoping it would be great for them. Am I just being too picky about the slight background noise? This is the most I've spent on a source so far so I was hoping it wouldn't have anything wrong with it...

Don't have issues with my Rag 1 or the two Rag 2's I've used re: speakers or headphones - in any gain. YMMV