Do you have any references or blind test results to prove this?
All outputs whether "headphone" or "line" are amplified, or else it would have the volume of a turntable connected directly to an amplifier without a phono stage in between. Funny how we lived with "double amping" for decades and now suddenly it's the bane of the portable audio world. I promise you that if you use a voltmeter and test tone to match the headphone out to the line out, you will never be able to hear a difference.
Just to clear things up, the amp is not simply a volume pot. If you look at the hifi world, the volume pot is on the pre amp, which gives the gain as well as volume control. In head Fi, the pre amp and power amp are combined in an integrated amp form.
What does the power amp do? Essentially, the dac puts out anywhere from 1.5-2vrms of signal. This is a high voltage and with most iems will blow your brains out. Our integrated headphone amps have two purposes, firstly, volume control through he potentiometer. The other issue is that with a very small exceptional few, most DACs are designed to only output an extremely tiny current. Can it output a larger current? It can, but the signal starts to deteriorate. That is where the amp comes in. The amp has a very high input impedance in the range of 10k ohms. This allows the signal to be passed from the dac to the amp with minimal deterioration as the current which the dac has to put out is very small. The amplifier takes the signal, and puts out the signal to the iem/headphone which has a much lower resistance (usually between 32-300 ohms) the lower resistance results in a larger current for the same voltage (V=RI). The amplifier is able to provide this larger current without signal deterioration though as it is designed to do thus, the amp section is not simply a volume pot.
However, you are not wrong in saying that double amping is not as bad as people make it out to be even though you did get mixed up about what the amp is. Most amplifier sections are able to put out very good signals at very Low current levels, but as the current output rises the quality of the signal drops (albeit not as badly as with dacs). Based in the above equation, you can see that when you double amp, the the first amplifier (say the one on the m3) would be dealing with an extremely high resistance load putting out a very small current. Thus its effect on the end signal would not be as large as people make it out to be, and amping an amped signal can actually improve it.
If you try out the iBasso dx50 and 90, you'll realise that the volume control affects the line out too
Is here a difference between the sound of the line out and the headphone out, yes, definitely. It sounds different enough to be able to pick out without guessing. If you simply plug your headphones in, the difference will be very very stark. And the line out will sound pretty crappy most of the time as making the dac put out a large current like that causes massive signal deterioration. If you amp a line out and a headphone out though, differences become smaller. However, it is definitely still very audible.