You didn't persuade me. All you said was, "Run a test." That and, "I've run lots of tests." Have you ever actually tried HE6? It's OK to theorize, but I think you need to be upfront about the limits of your experience too.
But your tests haven't been bias controlled tests. My comments relate to any headphone and any amplifier. And I have no interest in persuading you of anything. My purpose is to help beginners to audio have better information in making decisions.
I'll provide a little background. In the late 1990's an audiophile group of which I was a member spent a couple of years doing bias controlled listening tests. These are comparisons between two products which are level matched and compared without the listener knowing which product is which. I apologize for the fact that we didn't document the tests but we did them for ourselves, not to prove anything to anybody. What we discovered were the same things that other had discovered by doing bias controlled listening tests.
One of things we learned is that modern, properly designed high fidelity solid state amplifiers are good enough now that none of them alter the sound they amplify to any audible extent. In order to get an amplifier to alter the signal audibly it has to be badly designed or have audible distortion like tube amps have, be defective or be used beyond its design parameters. We tested amplifiers ranging in price from $100 to $5000 and found audible differences only in tube amps and in amps that didn't measure well. Understand that we didn't test headphone amps because, in those days, headphone amps were built into receivers, preamps and integrated amps. There weren't many dedicated external headphone amps like we have today.
There is an audiophile myth that having unused reserve power affects sound quality. It is absolutely a myth. If it were true we would have found audible differences between powerful amps and weak amps that otherwise performed as a straight wire. As long as no units were stressed, this simply didn't happen. Neither price nor rated power affected sound quality.
Why does one buy a separate amplifier for headphones? I'm not sure but I can think of two good reasons. One would be to allow the user to play louder without distortion than the original amp. The other is to have a proper impedance match. Connecting a 50 ohm headphone to a 600 ohm output on a computer sound card, for instance, is not good. Amplifiers are good at dealing with loads that are higher in impedance than their own output impedance. They don't perform well when the load has a lower impedance. Amplifiers designed to drive speakers all have low output impedances because speakers generally present a lower impedance to the amp than a headphone. So if you are comparing that to an amp with a high output impedance then I would accept your findings.
If you hear better sound from the speaker terminals of a power amp than you do from something like a low impedance headphone amp with levels matched exactly, most likely you are being guided by hearing bias. Hearing bias is what causes people to hear audible differences from different copper wires. The differences are shortcuts their brains take when they cannot clearly discern an audible difference. It happens to everyone. When audible differences are clear, our brains deal with them normally. When audible differences are very subtle to non existent and we ask our ears to compare them, our brain takes shortcuts. It factors in visual inputs, expectation, preference etc to get the job done. It is an audible illusion in the same way optical illusions work and for the same reasons.
So here is a proper test for you. It takes at least 2 people. Set up a source that can run through two amplifiers at once. Match levels between the amplifiers to within a couple of millivolts at the headphone. No guessing. Use a digital multimeter on a test tone. Block the listener's view of what is going on. Play some music and have the person behind the blind switch from amp to amp randomly so that sometimes the same amp plays more than once in a row. With each switch ask the listener which amp is playing. Have the person behind the blind score the answers right or wrong. If the scores are close to 50-50 right vs. wrong then we say there is no audible difference. If the answers are 70% or more on either the right or wrong side, we say there is a clear audible difference. In between we would say the results are not clear. There may or may not be a subtle audible difference. Try it with different listeners. My bet is that, if you do it properly, there won't be an audible difference between the low impedance headphone amp and the speaker terminals on the power amp. It is boring, fussy and time consuming but it will resolve the issue as to whether the phenomenon is an audible difference or the result of bias.