or Connect
Head-Fi.org › Forums › Equipment Forums › Headphones (full-size) › How We Rank Our Own Headphones -- Statistics of 2,000 Owner Rankings, Compiled
New Posts  All Forums:Forum Nav:

How We Rank Our Own Headphones -- Statistics of 2,000 Owner Rankings, Compiled

post #1 of 56
Thread Starter 

We Rank Our Own Headphones - Statistics of 2,000 Owner Rankings, Compiled

 

SUMMARY: 

 

I have compiled an aggregated list of how we rank our own headphones, based on the thread "Rank the Headphones That you Own," ( http://www.head-fi.org/t/109756/rank-the-headphones-that-you-own#post_1284577 ).  Data is based on the most recent 2,000 rankings of that thread, extending approximately one year back.

 

This list differs from other lists in which a single reviewer compares multiple headphones.  This list aggregates the comparisons of many persons, each with some subset of the full list of headphone models.  It has been often said that the best headphone comparisons are those made by persons owning all headphones being compared:  this analysis attempts to capture these results.

 

This list is useful for someone considering a new headphone and wishes to know how folks who already have that headphone rank it against some others, with the hope that the inquirer will be familiar with some of the headphones to which it is compared.

 

To that end, I have:

  • Computed statistics on the various models and how many times each is ranked;
  • Developed an aggregated list of where each model falls, on the average, in a rank ordered list.  (Note that this is really not too useful, as each person ranks a given headphone against the others he or she owns, and each person ranking that headphone is ranking it against a different list);
  • Prepared a master ranking matrix, selecting only those headphones that appear on a ranking 10 or more times (to provide some statistical robustness), allowing each headphone of that population to be compared in ranking to all others (of that population), with indication of how many rankings are reflected for each comparison pair (and therefore with what degree of confidence each ranking has);
  • Performed some correlation plots that show where having one headphone model increases the likelihood of having certain other headphone models;
  • Set up the ability to provide, by request on this thread, ranking lists of any of the headphones ranked in the data set (even if they do not appear in the list posted of the most popular headphones... I can still search the database and post each comparison list that shows where that headphone is ranked by folks who own it against their other headphones).

 

GENERAL STATISTICS:

 

The 2,000 headphone rankings were extracted from posts on the thread "Rank the Headphones that you Own" with dates ranging from 5/31/2013 through 5/16/2014.  For folks who (endlessly) kept updating their ranking, I tried to use only their most recent ranking (though I admit some double rankings slipped through!).

 

Those 2,000 rankings encompass a total of 340 users posting their rankings.  The rankings included 816 headphone models.  However, 544 of these models only had one person ranking them.  Many of these were unique modifications to commercially-available headphones (e.g., "AKG Q701 w/ bass port sticker removed").  Others were truly one-of-a-kind users of apparently non-customized headphones (e.g. only one person ranked the HiFiMAN HE-300).  

 

Because the goal of this work was to provide some statistically reliable characterizations of how a particular headphone is ranked against others, I chose to concentrate extra attention on those headphone models that had at least 10 people ranking them. The following table identifies the 37 headphone models that met this criterion.  

 

Model Count Score
Sennheiser HD 800 53 8.9
HiFiMAN HE-6 18 8.2
Audeze LCD-3 13 8.2
HiFiMAN HE-500 35 8.1
Audeze LCD-2.2 23 7.9
Beyerdynamic T 1 24 7.7
Philips Fidelio X1 14 7.3
Fostex TH600 11 6.9
Sennheiser HD 600 46 7.1
Fostex TH900 13 6.9
Audeze LCD-2 16 6.5
Sennheiser HD 650 57 6.2
HiFiMAN HE-400 28 6.2
AKG Q701 17 6.0
Sennheiser HD 598 26 5.8
Beyerdynamic DT 990 12 5.7
Sennheiser Momentum 20 5.7
Beyerdynamic DT 880 16 5.5
Denon AH-D7000 12 5.4
Denon AH-D2000 19 5.4
AKG K550 13 5.4
AKG K701 17 5.4
Sennheiser HD 580 10 5.3
Beyerdynamic DT 1350 10 5.2
MrSpeakers Mad Dog V3.2 13 4.9
Sennheiser HD 25-1 II 10 4.8
Grado SR225i 12 4.8
Sennheiser Amperior 10 4.7
Shure SRH840 12 4.5
V-MODA M-100 10 4.3
Grado SR80 13 4.0
Grado SR80i 13 3.9
Audio-Technica ATH-M50 29 3.6
Koss KSC75 11 3.6
Sennheiser PX 100-II 10 3.4
Audio-Technica ATH-AD700 16 2.9
Koss Porta Pro 18 2.5

 

Fig. 1:  Listing of Headphone Models Ranked by 10 or More Persons, With Count Of Rankings And Positional Score (10=top rated).

 

The headphones each have a score ranging from 1 to 10, indicating (the reverse of) their average ranking across the persons who ranked them.  For example, if all 53 persons indicated in the "Count" column ranked the Sennheiser HD 800 at the top of their list, that headphone would have a score of 10.0.  The fact that it has a score of 8.9 shows that several of the 53 rankings placed other headphones above it.  In some cases, those were headphones on this list of 37 (e.g., HiFiMAN HE-6, Audeze LCD-3); in others, they were headphones not ranked often enough to appear on this list of 37 popular models (e.g., Stax SR-009, which had 6 persons ranking it).  

 

RANKINGS OF MOST POPULAR HEADPHONES:

 

The general score given in the above table is not as useful as knowing how a particular headphone was ranked against another particular headphone.  This section provides model-specific pairwise rankings.

 

All 37 headphones appear as both rows and columns in the following table.  Each cell shows the ranking of the two headphone models that label its row and its column position.  For the diagonal, both row and column model are identical and it is filled with a black square.  Below the diagonal, the information is simply reflected from above (i.e. ranking the Sennheiser HD 800 against the Beyerdynamic T1 is the same as ranking the T1 against the HD 800), so cells below the diagonal are left blank, since the same information exists in the table above the diagonal.

 

The color of the dot in the cell indicates who won the pairing... a blue or green dot indicates strong (blue) or medium (green) winning of the pair-off by the headphone model in the row title.  This is easy to remember because the row titles are blue.  Likewise, a red or orange dot indicates strong (red) or medium (orange) winning of the pair-off by the headphone model in the (conveniently red) column title at the top of the table.  If the two comparisons were approximately equal (e.g., 4 persons ranked the HiFiMAN HE-6 above the Sennheiser HD 800 and 6 persons ranked the HD 800 above the HE-6), the dot is white.

 

Specifically, the colors are assigned by the fraction of multiple rankings won by the (blue) row title as follows: 80%-100% (blue); 60% - 79% (green); 40% - 59% (white); 20% - 39% (orange), and 0% - 19% (red).  Hence, a red dot means the same for the red column model (more than 79% preferred) as a blue dot means for the blue column model (again, more than 79% preferred).

The confidence of each ranking is measured by the number of pair-wise comparisons that contributed to the dot.  If there were 10 or more, the background color of the cell is a dark gray; 5 to 9 gives a medium gray background, and 1 to 4 gives a light gray background.  If there were no instances of a ranking of one headphone model vs. another, the corresponding cell is blank (white, no dot).

 

 

 

Fig. 2:  Pairwise Ranking of 37 Most Popular Headphone Models (click to enlarge).

 

Hence, it is interesting to note that at least 10 persons ranked the Sennheiser HD 800 against the HiFiMAN HE-6, and approximately equal numbers selected either as superior to the other.  Hence, its cell is dark gray with a white dot.

 

CORRELATIONS OF HEADPHONE MODEL OWNERSHIP

 

Do "bassheads" tend to favor a certain set of headphone models in their purchases?  Are there "trebleheads" favoring a different set?  If an unusually large number of people that have a Sennheiser HD 600 also have a Sennheiser HD 650 (and the data shows that they do!), should you get one too to go with your HD 600?

 

This portion of the effort looked for pairs of headphone model ownership that were significantly higher than random chance.  The table below identifies where an unusually high number of persons that have one particular headphone model also have another headphone model.

 

The red cells show modest correlation (specifically, that the number of times someone ranks that pair of model pairs at least two standard deviations higher than a random distribution); the black cells show strong correlation (more than four standard deviations above random).

 

I need to look at these correlations more to find other effects, but I submit the data shows a "moneybags" effect with its heavy correlation in the stratosphere of top-ranking headphones at the upper left... people who have HD 800s also are more likely to have other top-end headphones such as the HiFiMAN HE-6 and Audeze LCD-3 (they probably have Stax SR-009 too, but not enough are in the data set to appear here).

 

 

 

Fig. 3:  Correlation of Headphone Models Owned:  Pink Cells Are Significant Correlations; Black Cells Are Highly Significant Correlations.

 

This sort of table is a start in automated recommender systems such as Netflix or Amazon, where user product preferences are predicted from their preferences of product they already have.

 

SPECIALIZED HEADPHONE PAIR COMPARISONS

 

This data is really a small subset of the information that is available.  If you are considering a headphone purchase or want to know how a headphone model fared in this data that did not appear here, just post to this thread and let me know, and I'll be glad to post both statistics and (if not too numerous) individual ranking lists (sometimes, such as for the HD 800, there are too many, 53 in that case).

 

For example, I have been pointed to the Beyerdynamic T 5 p as a great audiophile headphone for iPod portable use.  Only 6 folks ranked it, but I can retrieve the six ranking lists as shown here and see what headphones are ranked better and worse than the T 5 p (and by whom, if I want to ask questions).

 

0% 25% 55% 43% 0% 33%
Beyerdynamic T 5 p Sennheiser HD 800 Stax SR007 MK2 HiFiMAN HE-6 Beyerdynamic T 5 p Audeze LCD-2.2
Audeze LCD-3 Beyerdynamic T 5 p Sennheiser HD 800 Sennheiser HD 800 Audio-Technica ATH-W5000 Beyerdynamic T 5 p
Sennheiser HD 800 Sennheiser Amperior Audio-Technica ATH-ESW11LTD Beyerdynamic T 1 Sennheiser HD 650 Westone ES2
AKG K3003i AKG K701 HiFiMAN HE-6 Beyerdynamic T 5 p PSB M4U2 Beyerdynamic T 51 p
Ultrasone Edition 8 AKG K450 Audeze LCD-2.1 Grado GS1000i Sennheiser HD 380 PRO  
Shure E2c   Shure SE535 Audio-Technica W3000ANV    
    Beyerdynamic T 5 p Polk Ultrafocus 8000 (ANC)    
    AKG Q701 Logitech UE 900    
    Grado SR325i      
    Shure SE425      
    B&W P5      
    Sony XB 500      

 

Fig. 4: Table of the Six Rankings for the Beyerdynamic T 5 p - each ranking list is a column, with the number at the top giving the rank order position, in percent, of the model of interest (0% = top of list; 100% = bottom of list).  This is an example of the ranking lists that can be extracted for any headphone in the database.

 

Again, feel free to request individual model lists that I didn't already post here.

 

(edited: Jun 10, 2014 to reflect data omissions caught by jazzman7 (thanks!)... adds comparisons for HD 800 vs. HE-500, HE-50 vs. AKG K701, M-100 vs. PortaPro, PX 100-II vs. AD 700, AKG K701 vs Porta Pro, and Shure SRH840 vs. AD 700).


Edited by ruthieandjohn - 6/9/14 at 9:07pm
post #2 of 56

Wow....no NAD HP50's.

 

At the very least I'd have them above the X1's

post #3 of 56

Great post. May get more traffic in the regular headphone section though.

post #4 of 56
Thread Starter 

@Astropin,

 

NAD VISO hp50s were in the mix, but since there were only four ratings, they didn't make it into the 10-or-above cut.

 

Here are its four rankings (by the way, I have them and was Rater #4, on the far right):

 

0% 50% 33% 33%
NAD VISO HP50 Aedle VK-1 V-MODA M-100 Stax SR-5
HiFiMAN HE-400 w/ jergpads Sennheiser HD 25-1 II AKG K550 Grado PS500
Sennheiser HD 598 NAD VISO HP50 NAD VISO HP50 Sennheiser CX 300 II
Yamaha HPH-200 Monster Turbine Pro Gold Ultrasone PRO 900 Sennheiser HD 598
Koss Sportapro on KSC75 clips Denon DN-HP700 AKG Q701 NAD VISO HP50
Grado SR80   German Maestro GMP250 Beats Pro
AKG K240 DF   Westone ADV Beats Studio 2013
Koss KSC75 on Sportapro headband     Beats Studio 2013 Wireless
KRK KNS 8400     Parrot Zik
Beyerdynamic DT 231 Pro     Apple In Ear
      Klipsch S4i
      Apple Ear Pods
      Sennheiser PCX 310 BT

Edited by ruthieandjohn - 6/1/14 at 6:03pm
post #5 of 56

Thanks for going to all this effort. The results are quite fascinating.

post #6 of 56
Not sure it makes sense to separate:

LCD2 and LCD2.2
Q701, K701, and K702
SR80 and SR80i
KSC75 and Portapro
post #7 of 56

Incredible effort  :beerchug:

 

Out of curiosity I started to break down the average number of HP's owned in a given year (2002 10.4 and 2003 9.4) in the "Rank your headphones from 1-10 thread" it is a daunting and time consuming task.

 

When owners started adding HP's and subtracting HP's I just gave up.

 

The stats you compiled are amazing did you reach outside that thread or did you take it further?

post #8 of 56
Thread Starter 
Quote:
Originally Posted by Claritas View Post

Not sure it makes sense to separate:

LCD2 and LCD2.2
Q701, K701, and K702
SR80 and SR80i
KSC75 and Portapro


I understand.  I've seen some folks cite specific design or sonic differences between LCD-2 and LCD-2.2, and between Q701 and K701.  Not sure about SR80 and SR80i, and have NO familiarity with KSC75 and Porta Pro. 

 

But when I didn't know, I tried to err on the side of separation, the thought being that if someone changes a design to improve it, it will (hopefully) improve and sound better, or at least different (hey, I even separated out those common models where the owner helpfully offered that they had gone to genuine unobtainium cables to replace the braided sterling silver ones that were stock!).

 

Good observation!

post #9 of 56

since a kind of structure has been devised, maybe a regular half or yearly survey can be sent out to each member...

instead of waiting for postings to be made to the "Rank the Headphones that you Own" thread.

 

I dun see any stax... nor the AD, these happy campers aint shouting about these cans, just snuggling in with their whisky. :P

post #10 of 56
Thread Starter 
Quote:
Originally Posted by RUMAY408 View Post
 

Incredible effort  :beerchug:

 

Out of curiosity I started to break down the average number of HP's owned in a given year (2002 10.4 and 2003 9.4) in the "Rank your headphones from 1-10 thread" it is a daunting and time consuming task.

 

When owners started adding HP's and subtracting HP's I just gave up.

 

The stats you compiled are amazing did you reach outside that thread or did you take it further?


No, everything in the statistical analysis comes entirely from the "Rank the headphones that you own" thread, for the 1-year period of May 2013 through May 2014.  There is at least 4 times as much info prior to May 2013, but earlier I did a smaller analysis of just 1,000 entries from that thread and found that the data changed little when I went up to 2,000... for example 10 of the top 12 headphone models were the same in both analyses, with the Sennheiser Momentum and HiFiMAN HE-6 in the 2,000 model analysis displacing the AKG-K550 and the PortaPro in the 1,000 comparison analysis.

 

That trial compilation of the smaller data set is here:  http://www.head-fi.org/t/109756/rank-the-headphones-that-you-own/4065#post_10501387

 

Next steps could include:

 

  • Going further back in time... there are probably at least 5,000 comparisons in the whole thread, though some are multiple ones by the same person, updated - this would not change results much on the top 37 (in my opinion), but it would probably increase the number of models that have at least 10 comparisons to over 100 (and increase by a factor of 10 the number of analyses and spreadsheet cells to fill!!).
  • Looking at a different type of correlation... by user and how they actually rank order headphone combinations (right now I only look at whether they own various pairs of headphones, without regard to how highly they rank them)
  • Automate the spreadsheet so someone other than me could use it (for example, add a drop-down menu to choose a headphone model, rather than trying to remember whether one writes "Sennheiser HD 800" or Sennheiser HD-800" or "Sennheiser HD800." (I tried to write them how the manufacturer did on their own site, but of course folks doing ranking might just write "HD800" which complicates the database compares).
post #11 of 56

You did an incredible amount of hard work ^^

 

I love stats, and the HP's that are ranked are in the realm most can afford.

 

HP's $2,000 and above are beyond the scope of all but 1% or less of the population.

post #12 of 56
Thread Starter 
Quote:
Originally Posted by Lorspeaker View Post
 

since a kind of structure has been devised, maybe a regular half or yearly survey can be sent out to each member...

instead of waiting for postings to be made to the "Rank the Headphones that you Own" thread.

 

I dun see any stax... nor the AD, these happy campers aint shouting about these cans, just snuggling in with their whisky. :P


Ask and ye shall receive (fewer than 10 people ranked Stax, hence they are not in my original matrix).  Here is the Stax SR-009, which had far more rankings than the other flavors of Stax (SR-007, Lamda, etc, which all had either 1 or 2 persons).  Not sure what the "AD" is, but be glad to look for it if you tell me.

 

STAX SR-009 (6 rankings)

 

0% 0% 0% 0% 0% 0%
Stax SR-009 Stax SR-009 Stax SR-009 Stax SR-009 Stax SR-009 Stax SR-009
Sennheiser HD 800 Sennheiser HE 60 HiFiMAN HE-6 Fostex TH900 Stax SR-007 MkI  Audeze LCD-3
Audeze LCD-3 Sennheiser HD 800 Grado PS1000 Sennheiser HD 800 Airbow SR-SC21 Audeze LCD-X
Fostex TH900 Stax SR-007 Denon AH-D2000 Sennheiser HD 650 Audeze LCD-3 Sennheiser HD 800
Sennheiser HD 650 HiFiMAN HE-6 Audeze LCD-2 Audeze LCD-2 Sennheiser HD 800 MrSpeakers Alpha Dog
Sennheiser HD 600 Beyerdynamic T 1 Audio-Technica ATH-ESW11LTD Beyerdynamic T 1 Stax SR-007 MkII HiFiMAN HE-500
Beyerdynamic DT 880 250 ohm Audeze LCD-3 Stax Lambda PRO Sennheiser HD 600 Audeze LCD-2 Beyerdynamic DT 880 600 ohm
  Audio-Technica ATH-AD2000X Denon AH-D2000   Fostex TH900  
  Audeze LCD-2 JCV-FX700   Beyerdynamic T 1  
  Denon AH-D7000 Audio-Technica CK10   Sennheiser HD 650 recable  
    AKG K701   Beyerdynamic DT 880  
    Etymotic ER-6i   AKG K271 recable  
    Apple In Ear      
    Apple Ear Buds      
post #13 of 56

ADs are Mr. Speakers Alpha Dogs.

 

I'm guessing that Stax owners probably may not be aware of, and have little participation in the "Rank your headphones that you own" thread.

 

Good job on the analysis though.

post #14 of 56

wow...u are on top of this data ! :P 

 

AD...Mr Speakers Alpha Dog.

post #15 of 56
Thread Starter 
Quote:
Originally Posted by Lorspeaker View Post
 

wow...u are on top of this data ! :P 

 

AD...Mr Speakers Alpha Dog.


Thanks... 7 people ranked it - here you go:

 

25% 67% 25% 63% 80% 29% 67%
Sennheiser HD 800 Sennheiser HD 800 HiFiMAN HE-500 Stax Lambda 507 HiFiMAN HE-6 JVC DX1000 Stax SR-009
HiFiMAN HE-6 JVC DX1000 MrSpeakers Alpha Dog Sennheiser HD 800 Audeze LCD-3 Audeze LCD-2.2 Audeze LCD-3
MrSpeakers Alpha Dog HiFiMAN HE-500 Audio-Technica ATH-AD900 Fostex TH900 Sennheiser HD 800 MrSpeakers Alpha Dog Audeze LCD-X
Shure SE846 Audeze LCD-2.2 Beyerdynamic DT 880 Audeze LCD-2.2 Fostex TH600 Ultrasone Signature DJ Sennheiser HD 800
AKG K3003i Beyerdynamic T 1 Audio-Technica ATH-M50 Ultrasone Signature DJ MrSpeakers Alpha Dog Sennheiser HD 650 MrSpeakers Alpha Dog
Fostex T50RP Paradox Fostex TH600   MrSpeakers Alpha Dog Heir Audio 4.a Fostex TH600 HiFiMAN HE-500
Sennheiser Momentum MrSpeakers Alpha Dog   HiFiMAN HE-500   Audio Technical W1000 Beyerdynamic DT 880 600 ohm
Sennheiser Momentum On Ear Fischer Audio FA-011 Limited Edition   Grado RS1i + PS500   Grado Bushmills    
Monster Diamond Tears Grado Bushmills w/ Jumbo Pads   Heir Audio 4.ia      
  Sennheiser HD 600          
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Headphones (full-size)
Head-Fi.org › Forums › Equipment Forums › Headphones (full-size) › How We Rank Our Own Headphones -- Statistics of 2,000 Owner Rankings, Compiled