Some vintage (and modern) receivers, don't pay much attention to the headphone out - you can get pretty bad impedance mismatches, and other issues, depending on model and the headphones you want to use. So there *can* be a quality and power benefit.
For me, the benefit of a headphone amp is the size. I don't have room at my desk for a full size receiver. Also, many include fairly respectable DAC processors, which allows me some flexibility in source without requiring another box to process signals from my computer.
Also, they are fun - with choices of SS and different tube circuits, you can tailor the amp pretty well to your specific desires. And there is something to be said for using a product engineered for a specific end, rather than a subsidiary of its primary purpose.
But ultimately - if you have room for a full size receiver, and are happy with the sound you are getting - then that need vanishes. :)