Well you're actually looking at the DACs and not the amps here. I would expect DACs to have smaller differences than amps because they have the same goal (to convert a digital signal into an analog one), while amps can aim for specific power outputs to work better with specific types of headphones (and headphones will have the biggest impact on sound quality).
There are some pretty big power differences between any two "tier 2" Schiit amps (Asgard, Valhalla, and Lyr). Asgard is all solid state with good power, except into high impedances. Valhalla is all tube, so it offers much more power into high impedances at the expense of power into low impedance. Lyr offers even more power than Asgard and doesn't fall short into high impedances, but Valhalla does edge out an extra 140 mW into 300 Ohms.
Out of these three amps, Asgard is the only one, IMO, with a hard sell from a Schiit amp with a lower price (Magni). But as RRod (and Schiit) pointed out, the larger chassis/board offers more capability, giving you a gain switch and preamp outputs. Those would be pretty hard to add onto something with a 5" x 3.5" x 1.25" chassis.