a) Because there's an entire headphone amp industry that wants to perpetuate itself
b) Background noise levels when used with headphones
c) Volume control issues
d) Risk of blowing up drivers or eardrums if one's not careful.
There are some pretty good reasons why generally the average user is better/safer off getting a headphone amp imo. Of course if you know what you're doing and not afraid to experiment/own demanding cans then using a speaker amp *can* represent serious savings and better sound quality.