This post was incredibly refreshing bangraman, i want to thank you for that, along with the OP though as well, both very well written
And it's not even just about getting some sort of figures to discuss. As we've seen here, any idiot can use RMAA for example. The methodology to obtain the result is key: Good methodology is capable of eliminating, or greatly reducing irrelevant factors that might have an effect on your perception: You simply need to create a controlled methodology that has the least impact on an impartial outcome for any given situation.
Audio geekism and the enjoyment of product acquisition blinded me to this for the first couple of years after I got heavily into headphones / portable audio and also got back into the high-end audio in general, but since I apply this to large swathes of my purchasing, work and life it did come back with a vengeance in the end.
What about methodology? Take burn-in: All you have to do in terms of methodology is to buy two new headphones, check they are working the same at the outset, put them out in the same room (but not too close, as you don't want playback noise from one phone to affect the other passively) under dust covers and do an unattended burn-in of just one of them for ~100 hours in an environment that's not subject to large swings in temperature or humidity. If the differences are as profound as some people say they are, then as long as the above is carried out, in theory given the apparent large degree of the changes you shouldn't even need to heavily control the headphone swapping portion of the listening test - the difference should be very readily noticeable. Although obviously it pays to control the headphone-swapping portion of the test as well, i.e. for starters use a fast method of switching, and I don't mean fumbling with jacks, so that your audio memory remains unused as much as possible.
The fact that arguments about burn-in go on for ever in threads without anyone even touching on conducting, or having conducted the above test goes to show how far many people are from any degree of objectivity. I know that the outcome of this hobby of ours is an emotional experience, but to usefully discuss the merits of the devices involved there has to be some sort of quantifiable baseline. That is sorely lacking here oftentimes... and ultimately it can end up becoming a giant peer group(s) that for starters still allows some (even Head-Fi sponsoring) manufacturers to sell mediocre dreck, and do apparently very well out of it.
It's not simply about what you measure. It's about how you approach your opinions.