Another thing to add is contrary to common belief, an amplifier doesn't always "amplify" the signal therefore your logic is a bit wrong.
I can fully understand your viewpoint. "It's already loud enough, why would I want to make it louder?"
Therefore, very simplified consider an amplifier as a "regulator" of both voltage/current to be able to drive different loads (headphones) without distortion.


















