Mazuki quote-I think it is unfair to put DIY amps (M^3, PPA, Pimeta) in the same list as the commercial ones.
You are right regarding all the different configurations and permutations of DIY amps such as the M3, the PPA V1 and V2 and even the Pimeta. Your question is a good one that begs to be answered. When performing these tests how does one ensure a fair comparison?
I worked at a computer test lab in Los Angeles (right down the road from UCLA on Olympic Blvd.) and ran over 3000 projects. This question came up constantly when we did benchmark testing between competitive PC’s. We got into incredible battles over what was a “fair” configuration. If PC brand 1 beat brand 2 in a published test you can be sure I would have PC brand 2’s legal team call me to complain and demand a retraction because it was not an “apples to apples” fair and equal test. For example: PC brand 2 (the loser) had their top hardware “Evangelists” (aka gurus) argue that the hard drive had a faster seek time on PC brand 1 and that contributed to their margin of victory on intensive disk input/output tests. They were right.
We also did benchmark testing for AMD. If all motherboards were universal and (forward and backward) compatible with both Intel and AMD chips life would be easy and you could just benchmark the Intel, than change to the AMD. But life is rarely this easy (although it is in the case of changing compatible opamps even if you need a brown dog adapter).
The problem of similar headphone amp configurations is not only in the DIY domain. Look at the available configurations for one of the Singlepower amps:
Jensen Paper Oil Coupling
Audio Note Silver Foil Oil Coupling Capacitor
Solen Plate Voltage Input Capacitor
High Frequency Bypass Capacitors
Black Gate Power Capacitors
Black Gate Cathode Capacitors
Neutrik Locking Headphone Jack
Goldpoint Stepped Atten.
Some might argue that if you had an M3 with a stepped attenuater beat another amp with a standard ALPS volume control that the reason might be due (in part) to the advantage of the stepped attenuater.
I am open to ideas on how to make the comparisons better. We can provide stringent functional specification requirements for each amp tested.
1-Each amp must have the power supply on the same chassis as the amplifier circuit.
This would eliminate the less expensive wall wart power supplies of many amps. But this requirement would also mean we would not be able to test the ASL MG HEAD MKIII, which only comes with a separate power supply.
Conversely, if we specify:
2-Each amp will have the power supply offered with the base priced amplifier.
Would it be fair to test the Musical Fidelity X-Can V3 with the basic power supply against the Samuels Raptor? The same argument can be made for the Gilmore Lite. In this scenario we are penalizing the company that offers the option of a less expensive power supply and rewarding the one with no option.
What I tried to do is provide pertinent configuration information for the tested DIY amp samples, which is what I did with the M3 comparison. I did specify the opamp as 8610 and the power supply by Elpac. I know this amp can sound better with better parts because I have heard them and I greatly respect AMB and Morsel. But this was the configuration tested and (for now) all I can do is provide that information and let the readers decide.
If I had the time, I would rate the amps in a table with quantitative numbers for each permutation. In order to be fair you would have to do this for every amp and almost every key part and it becomes a bit unwieldy for the average user:
OPAMP 637 8
CAPACITORS BLACK GATES 9
POWER SUPPLY STEPS 9
HEADPHONE JACK NEUTRIK 8
OPAMP 8610 7
CAPACITORS PANASONIC FM 7
POWER SUPPLY ELPAC 6
HEADPHONE JACK CARDAS 7
I would than have to rank them as separate amps:
7-M3 STEPS WITH 637
14-M3 ELPAC WITH 8610
However, this only scratches the surface. What about tube configurations? Do you test with an AC power filter? What headphones do you use as your default? High impedance headphones will produce better sound with some amps. If we choose low impedance headphones as our single reference standard it will make many highly rated amps voiced for high impedance headphones sound less than optimal.
The list goes into the stratosphere and requires lots of work. My hat tips to Skylab for his effort.
For now, what I can do is list the specific test configuration. If I test another configurations (such as an M3 with 637s, STEPS, stepped attenuater etc.) I will rank it as a separate amp.