SanjWatsuki recently completed a survey of flagship headphone metrics replete with a grading system.
It's definitely a good idea to demand higher standards from flagship headphone manufacturers, however I think that the survey could use some fine tuning.
I'll go through a quick take of each of his criteria:
Let’s define what a flagship headphone should achieve.
- Bass linearity of +-5 dB from 20hz to 100hz.
- Bass linearity is difficult to achieve, but it should be done for a flagship.
- Even reference-grade dynamics tend to have a bass roll off. If the roll-off occurs below 40hz then even -10db could be acceptable. Foster dynamics (Ol' Denon DX000 & THXXX) go for the opposite mentality of adding a bass boost. There should be a criterion which determines the maximum extent of such a boost in relation to the midrange (200hz+). If the bass bleeds over too much then the effect can be pretty nasty.
- 100dB distortion should not exceed 0.8% beyond the sub-bass frequencies.
- No flagship should have any distortion that comes close to audibility.
- OK, arguable threshold through the midrange, less so for treble.
- 100dB distortion should not exceed 1% at 30hz.
- No flagship should have a messy bass due to distortion.
- A bit harsh on dynamics, maybe 3%?
- Frequency response curve should be very smooth with any resonances being very minor -- no major dips.
- A smooth frequency response suggests few resonances. Few resonances suggests a well engineered diaphragm and enclosure.
- Please add a definition of "major". Consider where the peaks/dips occur as well as the decay time of the resonances.
- Very small to absolutely no dip at 70hz-150hz.
- Those resonances are caused by an interaction of the headphone cushion and your face.
- Large bumps in the frequency response at those ranges suggests a poorly engineered headphone pad or that they didn’t do their testing with the human flesh as a variable.
- These bumps are usually ~3db, are they that important? If you are accounting for the human flesh variable then comfort should be of paramount importance, even at the cost of such a bump.
- Air-level treble should be no more than -15dB relative to the mid-range.
- A flagship should have excellent treble extension. Extreme treble roll-off should only exist on non-kilobuck headphones.
- Requires greater specificity and accounting for treble peaks (& the associated listening fatigue)
- No "wiggle" in the impedance graph.
- Wiggle suggests a poorly balanced voice coil in a dynamic headphone.
- Get that out of here if you’re a flagship.
- If the wiggle is not accompanied by a problematic resonance or major frequency response wiggle then who cares?
- Nearly perfect channel balance.
- If I’m paying over $1000 for a headphone, I expect the channel matching to be as good as it gets.
- Yes. 1db through the bass & midrange, 3db through treble.
- The headphone is open or semi-open. --- Arbitrary.
It's probably worth going into the intent behind the metrics.
My goal was to try and give points for what I believed to be a headphone rising to a technical challenge, rather than an absolute measurement of how good it sounds. Reproducing the lowest frequencies for dynamics is an example of one, and reproducing high frequencies for planar magnetics is another. You could create a headphone that would score very highly on this system without being a good headphone, for example. It's also worth noting that I came up with the metrics in a few minutes with arbitrary numbers responding to someone on this forum, actually. I just took the concept and ran with it for a few hours and out came this document.
This explains a few of the metrics. For example, the headphone cushion one -- if a company isn't willing to put time into the development of the cushion and its effects, then it shouldn't be a flagship. That technical challenge should be solved. Voice coil weirdness, likewise, should be solved. Even if the overall effect is minimal, I would expect that the company putting out the flagship to have given the headphone that much thought. When you're in the kilobuck range, I believe near perfection should be assumed and not giving that would be a massive red flag about possible other engineering failures in the headphone.
The criticisms about the numberless metrics are completely valid. A lot of them #4, #5, #7, and #8 really should have some sort of concrete definition. I didn't feel confident enough that I had a model that I could drop into #4, so I didn't. There are so few failures of #5 (i.e. DT48) that I wasn't sure where to put that threshold, I just thought "I'll play that one by ear." #7 likewise should have a concrete definition, as should #8. My biggest fault here is I really don't know where to put that threshold.
Frequency response analysis is hard. So much research is still going into it that I really don't feel safe making a strong comparison against any model yet. We have the Lorho modified DF, we have the Olive-Welti curve, there's an analysis for 1995 that proposed another curve, Sennheiser seems to have their own curve that almost all of their headphones match, as does Beyerdynamic, etc, etc. I tried to avoid that problem by trying to analyze the FR as minimally as I could, even though it probably is the most important metric of them all here, for this reason.
As for #9, I feel like openness is actually important and not arbitrary. Research is still going on as to the "why" open headphones perform better even with matched frequency response curves, but it's a fact that they score better. If we compare open vs. open and closed vs. closed, it's a bit arbitrary, but I feel there is a technical advantage that may not be possible to fix when compared open to closed. I feel like on a theoretical level you cannot make a closed headphone sound as good as an open headphone with the knowledge we have right now (at the flagship level).
----------------
Those criticisms in mind, let's open source the metrics to try and improve them.
1. The numbers originally given were arbitrary and I came up with them on the spot. +-5 dB just sounded nice at the time. Is a significant roll-off of the frequencies you feel, rather than hear, acceptable? I still feel like this should be relatively tight because most of the newer dynamic flagships appear to be trying to get great extension into the lowest frequencies -- I can't really think of a really high end dynamic flagship that doesn't shoot for this except for maybe the Grado HP1000.
2. Once again, arbitrary number was chosen on the spot. Let's open source the % and possibly break it up by treble and mid-range. Given more data, I wouldn't mind expanding this to tackle additional orders of distortion, but that data isn't comprehensive enough right now. Most headphones that don't have issues with odd resonances in the mid-range tend to fail this on the upper treble, so a split could make sense.
3. I feel like loosening this threshold is a good idea. Hearing a relatively low amount of distortion in those frequencies is incredibly tough -- harder than the rest of the frequency band. A greater separation makes sense.
4. It seems like the simplest way to analyze the FR for significant dips/peaks would be to have a reference curve to compare against. I hesitate to do that because of how in flux I think the research on FR curves is right now, but I can't think of a better solution off the top of my head. Anetode has suggested an analysis similar to Rin Choi's.
5. I still think this is a good one to keep around. We need to give this metric a concrete number, though. -3dB sounds as fine to me as any.
6. Not sure if we should tackle treble peaks in this one -- that sounds more like the domain of #4. I was mostly just trying to give credit for if a headphone managed to give good treble extension. Does anyone have a better concrete metric in mind for measuring treble extension?
7. I think this one is a good one to keep around but, like #5, we need to have a concrete measurement. It'd probably have to be by % of the impedance at 1khz or something. We'd also probably have to look at the other graphs to determine if the bump is actually a voice coil issue or if it was intentional or if it is just a housing resonance.
Tyll's article has some great info on an introduction to what we're trying to measure.
8. I agree that we need to have concrete measurements on this one. Anetode's suggestion of 1dB in bass and mid-range and 3dB in the treble seem like reasonable ballpark figures. I think we need to be a bit more exacting,though. I think we encounter measurement difficulties past 10khz, so we should probably only look up to 10khz. I feel like some of the IF measurements suffer from a poor seal, which also makes those ones hard to compare against. I also feel like this one SHOULD have unit-to-unit variation taken into account, but the data doesn't exist yet for most headphones.
9. Someone raised a good point about how a closed headphone shouldn't be penalized for being closed. There are closed headphones which can sound better than open headphones, after all. I can't think an elegant way to credit a closed headphone for that, though, and I don't personally think it exists at the flagship level yet. Does anyone have any ideas on this one?
