1. This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
    By continuing to use this site, you are consenting to our use of cookies.

    Dismiss Notice

How We Rank Our Own Headphones -- Statistics of 2,000 Owner Rankings, Compiled

Discussion in 'Headphones (full-size)' started by ruthieandjohn, Jun 1, 2014.
2 3 4
  1. ruthieandjohn
    We Rank Our Own Headphones - Statistics of 2,000 Owner Rankings, Compiled
    I have compiled an aggregated list of how we rank our own headphones, based on the thread "Rank the Headphones That you Own," ( http://www.head-fi.org/t/109756/rank-the-headphones-that-you-own#post_1284577 ).  Data is based on the most recent 2,000 rankings of that thread, extending approximately one year back.
    This list differs from other lists in which a single reviewer compares multiple headphones.  This list aggregates the comparisons of many persons, each with some subset of the full list of headphone models.  It has been often said that the best headphone comparisons are those made by persons owning all headphones being compared:  this analysis attempts to capture these results.
    This list is useful for someone considering a new headphone and wishes to know how folks who already have that headphone rank it against some others, with the hope that the inquirer will be familiar with some of the headphones to which it is compared.
    To that end, I have:
    1. Computed statistics on the various models and how many times each is ranked;
    2. Developed an aggregated list of where each model falls, on the average, in a rank ordered list.  (Note that this is really not too useful, as each person ranks a given headphone against the others he or she owns, and each person ranking that headphone is ranking it against a different list);
    3. Prepared a master ranking matrix, selecting only those headphones that appear on a ranking 10 or more times (to provide some statistical robustness), allowing each headphone of that population to be compared in ranking to all others (of that population), with indication of how many rankings are reflected for each comparison pair (and therefore with what degree of confidence each ranking has);
    4. Performed some correlation plots that show where having one headphone model increases the likelihood of having certain other headphone models;
    5. Set up the ability to provide, by request on this thread, ranking lists of any of the headphones ranked in the data set (even if they do not appear in the list posted of the most popular headphones... I can still search the database and post each comparison list that shows where that headphone is ranked by folks who own it against their other headphones).
    The 2,000 headphone rankings were extracted from posts on the thread "Rank the Headphones that you Own" with dates ranging from 5/31/2013 through 5/16/2014.  For folks who (endlessly) kept updating their ranking, I tried to use only their most recent ranking (though I admit some double rankings slipped through!).
    Those 2,000 rankings encompass a total of 340 users posting their rankings.  The rankings included 816 headphone models.  However, 544 of these models only had one person ranking them.  Many of these were unique modifications to commercially-available headphones (e.g., "AKG Q701 w/ bass port sticker removed").  Others were truly one-of-a-kind users of apparently non-customized headphones (e.g. only one person ranked the HiFiMAN HE-300).  
    Because the goal of this work was to provide some statistically reliable characterizations of how a particular headphone is ranked against others, I chose to concentrate extra attention on those headphone models that had at least 10 people ranking them. The following table identifies the 37 headphone models that met this criterion.  
    Sennheiser HD 800538.9
    HiFiMAN HE-6188.2
    Audeze LCD-3138.2
    HiFiMAN HE-500358.1
    Audeze LCD-2.2237.9
    Beyerdynamic T 1247.7
    Philips Fidelio X1147.3
    Fostex TH600116.9
    Sennheiser HD 600467.1
    Fostex TH900136.9
    Audeze LCD-2166.5
    Sennheiser HD 650576.2
    HiFiMAN HE-400286.2
    AKG Q701176.0
    Sennheiser HD 598265.8
    Beyerdynamic DT 990125.7
    Sennheiser Momentum205.7
    Beyerdynamic DT 880165.5
    Denon AH-D7000125.4
    Denon AH-D2000195.4
    AKG K550135.4
    AKG K701175.4
    Sennheiser HD 580105.3
    Beyerdynamic DT 1350105.2
    MrSpeakers Mad Dog V3.2134.9
    Sennheiser HD 25-1 II104.8
    Grado SR225i124.8
    Sennheiser Amperior104.7
    Shure SRH840124.5
    V-MODA M-100104.3
    Grado SR80134.0
    Grado SR80i133.9
    Audio-Technica ATH-M50293.6
    Koss KSC75113.6
    Sennheiser PX 100-II103.4
    Audio-Technica ATH-AD700162.9
    Koss Porta Pro182.5

    Fig. 1:  Listing of Headphone Models Ranked by 10 or More Persons, With Count Of Rankings And Positional Score (10=top rated).
    The headphones each have a score ranging from 1 to 10, indicating (the reverse of) their average ranking across the persons who ranked them.  For example, if all 53 persons indicated in the "Count" column ranked the Sennheiser HD 800 at the top of their list, that headphone would have a score of 10.0.  The fact that it has a score of 8.9 shows that several of the 53 rankings placed other headphones above it.  In some cases, those were headphones on this list of 37 (e.g., HiFiMAN HE-6, Audeze LCD-3); in others, they were headphones not ranked often enough to appear on this list of 37 popular models (e.g., Stax SR-009, which had 6 persons ranking it).  
    The general score given in the above table is not as useful as knowing how a particular headphone was ranked against another particular headphone.  This section provides model-specific pairwise rankings.
    All 37 headphones appear as both rows and columns in the following table.  Each cell shows the ranking of the two headphone models that label its row and its column position.  For the diagonal, both row and column model are identical and it is filled with a black square.  Below the diagonal, the information is simply reflected from above (i.e. ranking the Sennheiser HD 800 against the Beyerdynamic T1 is the same as ranking the T1 against the HD 800), so cells below the diagonal are left blank, since the same information exists in the table above the diagonal.
    The color of the dot in the cell indicates who won the pairing... a blue or green dot indicates strong (blue) or medium (green) winning of the pair-off by the headphone model in the row title.  This is easy to remember because the row titles are blue.  Likewise, a red or orange dot indicates strong (red) or medium (orange) winning of the pair-off by the headphone model in the (conveniently red) column title at the top of the table.  If the two comparisons were approximately equal (e.g., 4 persons ranked the HiFiMAN HE-6 above the Sennheiser HD 800 and 6 persons ranked the HD 800 above the HE-6), the dot is white.
    Specifically, the colors are assigned by the fraction of multiple rankings won by the (blue) row title as follows: 80%-100% (blue); 60% - 79% (green); 40% - 59% (white); 20% - 39% (orange), and 0% - 19% (red).  Hence, a red dot means the same for the red column model (more than 79% preferred) as a blue dot means for the blue column model (again, more than 79% preferred).
    The confidence of each ranking is measured by the number of pair-wise comparisons that contributed to the dot.  If there were 10 or more, the background color of the cell is a dark gray; 5 to 9 gives a medium gray background, and 1 to 4 gives a light gray background.  If there were no instances of a ranking of one headphone model vs. another, the corresponding cell is blank (white, no dot).
    Fig. 2:  Pairwise Ranking of 37 Most Popular Headphone Models (click to enlarge).
    Hence, it is interesting to note that at least 10 persons ranked the Sennheiser HD 800 against the HiFiMAN HE-6, and approximately equal numbers selected either as superior to the other.  Hence, its cell is dark gray with a white dot.
    Do "bassheads" tend to favor a certain set of headphone models in their purchases?  Are there "trebleheads" favoring a different set?  If an unusually large number of people that have a Sennheiser HD 600 also have a Sennheiser HD 650 (and the data shows that they do!), should you get one too to go with your HD 600?
    This portion of the effort looked for pairs of headphone model ownership that were significantly higher than random chance.  The table below identifies where an unusually high number of persons that have one particular headphone model also have another headphone model.
    The red cells show modest correlation (specifically, that the number of times someone ranks that pair of model pairs at least two standard deviations higher than a random distribution); the black cells show strong correlation (more than four standard deviations above random).
    I need to look at these correlations more to find other effects, but I submit the data shows a "moneybags" effect with its heavy correlation in the stratosphere of top-ranking headphones at the upper left... people who have HD 800s also are more likely to have other top-end headphones such as the HiFiMAN HE-6 and Audeze LCD-3 (they probably have Stax SR-009 too, but not enough are in the data set to appear here).
    Fig. 3:  Correlation of Headphone Models Owned:  Pink Cells Are Significant Correlations; Black Cells Are Highly Significant Correlations.
    This sort of table is a start in automated recommender systems such as Netflix or Amazon, where user product preferences are predicted from their preferences of product they already have.
    This data is really a small subset of the information that is available.  If you are considering a headphone purchase or want to know how a headphone model fared in this data that did not appear here, just post to this thread and let me know, and I'll be glad to post both statistics and (if not too numerous) individual ranking lists (sometimes, such as for the HD 800, there are too many, 53 in that case).
    For example, I have been pointed to the Beyerdynamic T 5 p as a great audiophile headphone for iPod portable use.  Only 6 folks ranked it, but I can retrieve the six ranking lists as shown here and see what headphones are ranked better and worse than the T 5 p (and by whom, if I want to ask questions).
    Beyerdynamic T 5 pSennheiser HD 800Stax SR007 MK2HiFiMAN HE-6Beyerdynamic T 5 pAudeze LCD-2.2
    Audeze LCD-3Beyerdynamic T 5 pSennheiser HD 800Sennheiser HD 800Audio-Technica ATH-W5000Beyerdynamic T 5 p
    Sennheiser HD 800Sennheiser AmperiorAudio-Technica ATH-ESW11LTDBeyerdynamic T 1Sennheiser HD 650Westone ES2
    AKG K3003iAKG K701HiFiMAN HE-6Beyerdynamic T 5 pPSB M4U2Beyerdynamic T 51 p
    Ultrasone Edition 8AKG K450Audeze LCD-2.1Grado GS1000iSennheiser HD 380 PRO 
    Shure E2c Shure SE535Audio-Technica W3000ANV  
      Beyerdynamic T 5 pPolk Ultrafocus 8000 (ANC)  
      AKG Q701Logitech UE 900  
      Grado SR325i   
      Shure SE425   
      B&W P5   
      Sony XB 500   

    Fig. 4: Table of the Six Rankings for the Beyerdynamic T 5 p - each ranking list is a column, with the number at the top giving the rank order position, in percent, of the model of interest (0% = top of list; 100% = bottom of list).  This is an example of the ranking lists that can be extracted for any headphone in the database.
    Again, feel free to request individual model lists that I didn't already post here.
    (edited: Jun 10, 2014 to reflect data omissions caught by jazzman7 (thanks!)... adds comparisons for HD 800 vs. HE-500, HE-50 vs. AKG K701, M-100 vs. PortaPro, PX 100-II vs. AD 700, AKG K701 vs Porta Pro, and Shure SRH840 vs. AD 700).
  2. Astropin
    Wow....no NAD HP50's.
    At the very least I'd have them above the X1's
  3. Stuff Jones
    Great post. May get more traffic in the regular headphone section though.
  4. ruthieandjohn
    NAD VISO hp50s were in the mix, but since there were only four ratings, they didn't make it into the 10-or-above cut.
    Here are its four rankings (by the way, I have them and was Rater #4, on the far right):
    NAD VISO HP50Aedle VK-1V-MODA M-100Stax SR-5
    HiFiMAN HE-400 w/ jergpadsSennheiser HD 25-1 IIAKG K550Grado PS500
    Sennheiser HD 598NAD VISO HP50NAD VISO HP50Sennheiser CX 300 II
    Yamaha HPH-200Monster Turbine Pro GoldUltrasone PRO 900Sennheiser HD 598
    Koss Sportapro on KSC75 clipsDenon DN-HP700AKG Q701NAD VISO HP50
    Grado SR80 German Maestro GMP250Beats Pro
    AKG K240 DF Westone ADVBeats Studio 2013
    Koss KSC75 on Sportapro headband  Beats Studio 2013 Wireless
    KRK KNS 8400  Parrot Zik
    Beyerdynamic DT 231 Pro  Apple In Ear
       Klipsch S4i
       Apple Ear Pods
       Sennheiser PCX 310 BT

  5. Currawong Contributor
    Thanks for going to all this effort. The results are quite fascinating.
    Admiral Kolchak likes this.
  6. Claritas
    Not sure it makes sense to separate:

    LCD2 and LCD2.2
    Q701, K701, and K702
    SR80 and SR80i
    KSC75 and Portapro
    Harry Manback likes this.
  7. RUMAY408
    Incredible effort  [​IMG]
    Out of curiosity I started to break down the average number of HP's owned in a given year (2002 10.4 and 2003 9.4) in the "Rank your headphones from 1-10 thread" it is a daunting and time consuming task.
    When owners started adding HP's and subtracting HP's I just gave up.
    The stats you compiled are amazing did you reach outside that thread or did you take it further?
  8. ruthieandjohn

    I understand.  I've seen some folks cite specific design or sonic differences between LCD-2 and LCD-2.2, and between Q701 and K701.  Not sure about SR80 and SR80i, and have NO familiarity with KSC75 and Porta Pro. 
    But when I didn't know, I tried to err on the side of separation, the thought being that if someone changes a design to improve it, it will (hopefully) improve and sound better, or at least different (hey, I even separated out those common models where the owner helpfully offered that they had gone to genuine unobtainium cables to replace the braided sterling silver ones that were stock!).
    Good observation!
    Admiral Kolchak likes this.
  9. Lorspeaker
    since a kind of structure has been devised, maybe a regular half or yearly survey can be sent out to each member...
    instead of waiting for postings to be made to the "Rank the Headphones that you Own" thread.
    I dun see any stax... nor the AD, these happy campers aint shouting about these cans, just snuggling in with their whisky. :p
  10. ruthieandjohn

    No, everything in the statistical analysis comes entirely from the "Rank the headphones that you own" thread, for the 1-year period of May 2013 through May 2014.  There is at least 4 times as much info prior to May 2013, but earlier I did a smaller analysis of just 1,000 entries from that thread and found that the data changed little when I went up to 2,000... for example 10 of the top 12 headphone models were the same in both analyses, with the Sennheiser Momentum and HiFiMAN HE-6 in the 2,000 model analysis displacing the AKG-K550 and the PortaPro in the 1,000 comparison analysis.
    That trial compilation of the smaller data set is here:  http://www.head-fi.org/t/109756/rank-the-headphones-that-you-own/4065#post_10501387
    Next steps could include:
    1. Going further back in time... there are probably at least 5,000 comparisons in the whole thread, though some are multiple ones by the same person, updated - this would not change results much on the top 37 (in my opinion), but it would probably increase the number of models that have at least 10 comparisons to over 100 (and increase by a factor of 10 the number of analyses and spreadsheet cells to fill!!).
    2. Looking at a different type of correlation... by user and how they actually rank order headphone combinations (right now I only look at whether they own various pairs of headphones, without regard to how highly they rank them)
    3. Automate the spreadsheet so someone other than me could use it (for example, add a drop-down menu to choose a headphone model, rather than trying to remember whether one writes "Sennheiser HD 800" or Sennheiser HD-800" or "Sennheiser HD800." (I tried to write them how the manufacturer did on their own site, but of course folks doing ranking might just write "HD800" which complicates the database compares).
    Admiral Kolchak likes this.
  11. RUMAY408
    You did an incredible amount of hard work ^^
    I love stats, and the HP's that are ranked are in the realm most can afford.
    HP's $2,000 and above are beyond the scope of all but 1% or less of the population.
    Admiral Kolchak likes this.
  12. ruthieandjohn

    Ask and ye shall receive (fewer than 10 people ranked Stax, hence they are not in my original matrix).  Here is the Stax SR-009, which had far more rankings than the other flavors of Stax (SR-007, Lamda, etc, which all had either 1 or 2 persons).  Not sure what the "AD" is, but be glad to look for it if you tell me.
    STAX SR-009 (6 rankings)
    Stax SR-009Stax SR-009Stax SR-009Stax SR-009Stax SR-009Stax SR-009
    Sennheiser HD 800Sennheiser HE 60HiFiMAN HE-6Fostex TH900Stax SR-007 MkI Audeze LCD-3
    Audeze LCD-3Sennheiser HD 800Grado PS1000Sennheiser HD 800Airbow SR-SC21Audeze LCD-X
    Fostex TH900Stax SR-007Denon AH-D2000Sennheiser HD 650Audeze LCD-3Sennheiser HD 800
    Sennheiser HD 650HiFiMAN HE-6Audeze LCD-2Audeze LCD-2Sennheiser HD 800MrSpeakers Alpha Dog
    Sennheiser HD 600Beyerdynamic T 1Audio-Technica ATH-ESW11LTDBeyerdynamic T 1Stax SR-007 MkIIHiFiMAN HE-500
    Beyerdynamic DT 880 250 ohmAudeze LCD-3Stax Lambda PROSennheiser HD 600Audeze LCD-2Beyerdynamic DT 880 600 ohm
     Audio-Technica ATH-AD2000XDenon AH-D2000 Fostex TH900 
     Audeze LCD-2JCV-FX700 Beyerdynamic T 1 
     Denon AH-D7000Audio-Technica CK10 Sennheiser HD 650 recable 
      AKG K701 Beyerdynamic DT 880 
      Etymotic ER-6i AKG K271 recable 
      Apple In Ear   
      Apple Ear Buds   

  13. AnakChan Moderator
    ADs are Mr. Speakers Alpha Dogs.
    I'm guessing that Stax owners probably may not be aware of, and have little participation in the "Rank your headphones that you own" thread.
    Good job on the analysis though.
  14. Lorspeaker
    wow...u are on top of this data ! :p 
    AD...Mr Speakers Alpha Dog.
  15. ruthieandjohn

    Thanks... 7 people ranked it - here you go:
    Sennheiser HD 800Sennheiser HD 800HiFiMAN HE-500Stax Lambda 507HiFiMAN HE-6JVC DX1000Stax SR-009
    HiFiMAN HE-6JVC DX1000MrSpeakers Alpha DogSennheiser HD 800Audeze LCD-3Audeze LCD-2.2Audeze LCD-3
    MrSpeakers Alpha DogHiFiMAN HE-500Audio-Technica ATH-AD900Fostex TH900Sennheiser HD 800MrSpeakers Alpha DogAudeze LCD-X
    Shure SE846Audeze LCD-2.2Beyerdynamic DT 880Audeze LCD-2.2Fostex TH600Ultrasone Signature DJSennheiser HD 800
    AKG K3003iBeyerdynamic T 1Audio-Technica ATH-M50Ultrasone Signature DJMrSpeakers Alpha DogSennheiser HD 650MrSpeakers Alpha Dog
    Fostex T50RP ParadoxFostex TH600 MrSpeakers Alpha DogHeir Audio 4.aFostex TH600HiFiMAN HE-500
    Sennheiser MomentumMrSpeakers Alpha Dog HiFiMAN HE-500 Audio Technical W1000Beyerdynamic DT 880 600 ohm
    Sennheiser Momentum On EarFischer Audio FA-011 Limited Edition Grado RS1i + PS500 Grado Bushmills   
    Monster Diamond TearsGrado Bushmills w/ Jumbo Pads Heir Audio 4.ia   
     Sennheiser HD 600     

2 3 4

Share This Page