I agree with scrypt. To borrow psychological terminology, a flat affect disturbs me far more than anything else. There is something infinitely dehumanizing about being on the receiving end of a blank, emotionless stare. The connection that all humans feel, and - to some extent - desire, is not there. If anything, there is an implied disregard for humanity in such an appearance; i.e., there is no empathy. That is real horror. For example, to use Lovecraft, Cthulhu operates beyond human ideas of good and evil. Pity isn't part of the Old One's emotional grammar. That is terror.
At the risk of being melodramatic, when one sees a vacant, expressionless countenance, one sees the abyss. Nothingness is far more terrifying than a babbling madman with a cleaver. The madman can be restrained; seeing the void reminds one that being "human" isn't necessarily an unalienable condition.