*work in progress* My Evaluation Methods // Glossary of Terms I Use // FR Graphs for Newbs

[tinyman392]

Another proposition for you Inks...  Before you send off the IEM to get measured, open up a parametric EQ and "draw" what you hear the graph to look like.  Compare afterwards.  Remember, ears don't always hear the way the machine will.  No matter how much the mechanism to measure uses a natural canal, your ear will always differ.  It creates a visual out of your subjective words.  Of course you should also post the objectively measured graph as well.  It'd also make a nice comparison. 
 
Remember, what you hear won't always be = to what is measured.  It's important to get both. 

 

[Inks]

  Answers in bold

There's still interpretation of graphs and that's not an absolute. Good stuff and Rin's work is fab. I do understand the desire to have a means for an irrefutable result but I don't think it actually exists. I think when a graph has a significant aberration, then it's telling you something but minor fluctuations in steady state testing can only say so much. I think a graph can bias as easily as reveal and reading in is not beyond human nature. One way around it is to do the subjective portion without ever looking at or discussing a measurement. I think most would corroberate but I bet there would be some that don't. Not to a great degree but rankings could vary a bit.
 
To some extent yes, there may be subjective interpretation. For the most part though, it's going to be as cold as evaluating performance in relation as to how close it gets to the diffuse-field curve. Like tinyman mentioned, I may draw my own curves based on parametric EQs before they are sent for measuring. Like I mention, Rin's simulator is based on the average ear, my ear's resonance may be slightly different. Still because Rin's is the average, I expect those to be closer to other individuals rather than myself. There may also be an instance where I may measure just how close my ears are to the simulator to the best method I could think of, but it's mostly just a plan for the time being. 
 
Of course a graph that showed bass you didn't hear would have you re-examine the listening portion and that would be a good thing. Just an example as I know that not getting a seal wouldn't be an issue for you.
 
Yeah, as for seal, whenever I feel the IEM may require a shallow seal (see Rin's Yamaha review), I let Rin graph them in that configuration. Luckily, I do get a good seal with most IEMs and have no issue with deep insertions using tips I have "broken-in" like my Klipsch ovals. 
 
That said, I'm looking forward to this and Rin's graphs as I do find them fascinating and extremely well done. I really enjoy them and wish you well on this endeavor. It's as good a method as any. It's not like we can help being human, even if we try.
 
I do have a question. What kit will be used to audition? I always like when a few items are used so that any particular electrical or sonic interface is not too prevalent.
I will simply use my iPhone 4, maybe an O2 amp, but mostly just the iPhone. I do this because I feel IEMs should be evaluated in regards to their portability, the iPhone doesn't have any major issues in sound, it's pretty transparent itself so I feel I have good grounds. 

 
 

There are some things that can't be done with an FR chart though:

• Decay speed/time - This must be done with a spectral decay graph
• Detailing - Detailing cannot be shown through an FR graph 100% of the time.  Although detailing in a certain range is more likely when the range is forward, finding out if it'll detail if the range is recessed or backgrounded is still uncertain.  Many V-shape IEMs do mid-range detailing extremely well.  Take the PFE232 for example.
• Distortion - There are other graphs that can show this though
• Timbre - Timbre is such an intricate and delicacy that it can't be found on an FR graph.  It would deal with the same ideas as the rest of the properties, but since timbre isn't a specific area, rather, it's all of them and in different quantities. 
• Clarity - Clarity cannot be shown directly on an FR 100% of the time.  It is very similar to detailing...  Forward normally means good clarity, but recessed becomes a mixed bag.
• Sibilance - Like detailing/clarity, sibilance cannot be shown directly on an FR 100% of the time.  Forward highs with big troughs/peaks can mean sibilant, but doesn't have to (PFE232 is a prime example), recessed with small troughs/peaks can be sibilant at times as well. 

 
 I am going to use SD  and Impulse response graphs as well, need to edit my post to mention it. Detailing can be shown 100% on the graph, the problem is that the term itself is far too broad. Our brains may fixate on what is details overlooking others. IME the more v-shaped the IEM the less detailed overall, yes you can have a fairly decently detailed IEM that is v-shaped but it's when you compare to a reference than the veils come into light. Timbre for the most part yes can be graphed, since overall tonal color is measured. The FX700s timbre is noticeably altered by the wooden diaphragm, that can't be measured, but it's tone isn't that well done unfortunately and it shows on FR. For clarity it isn't about having things forward but balanced, especially the bass, too much 200-300 is going to rid of clarity regardless of other forward frequencies. Sibilance lies in specific 4-8k regions, having a big bulk of it's acoustic energy at 4k but can reach up to 8k. Yes an IEM can be pretty mellow in the treble but may still sound sibilant, this happens because a peak has a occurred, easily shown on a FR graph. PFE232s will have to be tested by me for me to draw conclusions about it, but if it's sibilant there's likely to be a peak. I do need to mention dynamic range, that can't be measured and I will be using this method to test it. 
 
There are others as well, these are just 6 of them that are shown.  Many properties like texture can be shown if it's forward.  But if it's recessed on the FR graph, it can still have texture.  The forward areas are generally a "confirmation", but the lower areas are the ones where you may want to look into and listen for.  An FR graph can be used to confirm all subjective findings however as well as the other way around (all subjective findings can confirm an FR graph). They should be used in harmony for a really strong review. 
 
Can't help but feel that I'm getting a lot of guesses here, yes an IEM with a dip in it's texture region can still have texture, likely because it's coming from a boost is occurring after. Extent of properties can be relative to the listener, that's something to consider. For my attempt, I'm going to try to make it more exact. 

Likely to make my own graph for the A161P review based on EQing, but readers have to consider that it's going to be really estimated and I don't expect it to be as accurate as I would like it, not having the resources to make it more exact. 

 

[goodvibes]

I just thought of something Rin could measure the next time he does. I suspect that IEMs are held in place by their tips for measurement but some would normally ground or even wedge against the tragus. I'm wondering if any measurement would change if a bit of sorbothane were used mimick that in the dummy head. May vary by tip stiffness and driver alignment etc. or not at all.

 

[eke2k6]

I see what you're trying to do here Inks, and it's a mighty undertaking. 
 
However, I believe you're getting carried away with the graphs. All a graph will tell you is how much of something is there, and no more. In my mind, your search for 'the ideal' is doomed to fail from the start for the fact that it simply does not exist. You argue for phones that achieve the true intention of the producer instead of enjoying the music, yet you fail to account for the most important variable in the equation: me and you. Music is supposed to make you feel, not think. I had some time with the Etymotic HF5 last month, and I found it to be a spectacular IEM. However, while A/Bing with my ASG-1, I noticed that it couldn't quite produce violins with that rich timbre it should. My undergrad college had a VERY good music program, where I participated in music productions along with cellists, violinists, and pianists, so I know how these things sound. The HF5, despite it's great FR graph, simply couldn't pump out the euphoric sound that my ASG-1 did on Yo-Yo Ma's Goat Rodeo Sessions album. Another example is the GR07, which is notorious for it's treble spike. This intentional spike makes percussion and cymbals come alive in songs. I'm sure that removing that spike would render the iem less special as a whole. The FX700 is also a flawed IEM by your definition. Yet it is also one of the most highly rated earphones around, and so are most members of the Final Audio lineup.
 
My point is that graphs are simply a part of the puzzle. It's like the political debate in the US where one side wants to cut taxes and the other wants to raise them. No one solution will work, but a cocktail of ideas will help paint a better picture. The same can be said about audio.
 
The beauty of joker's reviews is that he marries SQ objectivity with subjectivity in a nearly universal language. These two MUST come together to successfully convey what an IEM is all about.

 

[Inks]

I just thought of something Rin could measure the next time he does. I suspect that IEMs are held in place by their tips for measurement but some would normally ground or even wedge against the tragus. I'm wondering if any measurement would change if a bit of sorbothane were used mimick that in the dummy head. May vary by tip stiffness and driver alignment etc. or not at all.

   I'll forward this to him. 

I see what you're trying to do here Inks, and it's a mighty undertaking. 
 
However, I believe you're getting carried away with the graphs. All a graph will tell you is how much of something is there, and no more. In my mind, your search for 'the ideal' is doomed to fail from the start for the fact that it simply does not exist. You argue for phones that achieve the true intention of the producer instead of enjoying the music, yet you fail to account for the most important variable in the equation: me and you. Music is supposed to make you feel, not think. I had some time with the Etymotic HF5 last month, and I found it to be a spectacular IEM. However, while A/Bing with my ASG-1, I noticed that it couldn't quite produce violins with that rich timbre it should. My undergrad college had a VERY good music program, where I participated in music productions along with cellists, violinists, and pianists, so I know how these things sound. The HF5, despite it's great FR graph, simply couldn't pump out the euphoric sound that my ASG-1 did on Yo-Yo Ma's Goat Rodeo Sessions album. Another example is the GR07, which is notorious for it's treble spike. This intentional spike makes percussion and cymbals come alive in songs. I'm sure that removing that spike would render the iem less special as a whole. The FX700 is also a flawed IEM by your definition. Yet it is also one of the most highly rated earphones around, and so are most members of the Final Audio lineup.
 
  Like I have said many times, there is no perfect or ideal, that misses the point. It's about how true to the nature of the recording the IEM gets, the Phonak recently measured is a prime example. Is it perfectly flat? No, but it's pretty darn close, making it an outstanding performer, too bad the housing design goes against the sound intended as mentioned by Rin. 
 
Anyway, the level of enjoyment for the individual and "feel" is more in the subjective realm, it will just make things even less tangible, that's something for the individual to conclude, can't share preferences because they're all over the place. I do enjoy my music very much, I don't try to look for details for casual listening, but that's beside the point. Knowing how a violin sounds like doesn't mean you'll specifically know how that violin recorded truly sounded in it's recording nature, unless you've perhaps have heard a reference loudspeaker set-up of that recording. I will have to check out the GR07's spike myself, IME with them, it seems to make it truer to the source despite risking harshness compared to a tamer presentation. FX700s go against a lot of how a recording was intended to sound, but it does sound special due to  the wooden diaphargm and soundstage. It's a nice product but I wouldn't call it an outstanding performer, it falls flat at times because of the excessive bass and treble boosts.  I feel a lot of these highly priced IEMs are rated highly because of their prestige not performance, the preferences of certain users shouldn't weight highly on accuracy which is the criteria here . I recently got the ER-6, released 9 years ago. It is notably more realistic  than the newer, much more expensive EX1000, FAD S, C, SB and a bunch of other highly priced IEMs you can view in my IEM history list.
 
My point is that graphs are simply a part of the puzzle. It's like the political debate in the US where one side wants to cut taxes and the other wants to raise them. No one solution will work, but a cocktail of ideas will help paint a better picture. The same can be said about audio.
 
The beauty of joker's reviews is that he marries SQ objectivity with subjectivity in a nearly universal language. These two MUST come together to successfully convey what an IEM is all about.
The more subjectivity, the less tangible the rating systems are going to be, subjective ratings are good for personal preference, that's it. A universal rating system needs to have set rules and guidelines, with tools to measure data in the most objective way possible. Don't forget, that this isn't a 100% objective approach, subjectivity is there, it's inevitable, so there's a mix of both, but the objective needs a bigger stance to create a more grounded system. 

 

[tinyman392]

Inks...  The Phonaks (232) have a 8-10dB dip in the midrange (Grey filters...  Get's deeper with others)...  I do have a graph (looks raw) that Phonak provided.  I can't publish it publicly since it was never public material (and it's copyrighted most likely) and I don't have distribution rights (from what I'm aware of). 
 
The rest of the Phonak series isn't as flat either...
 

​

 
There's a nice 4-5 dB V in the midrange with the black filters, even larger with the green (greens veil, blacks as well a little).  The Greys have a nice large spike in the 2.7 area (I know why this is) and have spikes in the rest of the treble (can't explain this one). 
 
The quality of the 232s beats out that of the 1xx series with grey filters in everything except for clarity...  Flatter isn't always better... This is why I stick with the statement that flatter isn't always better...  It's more than that.  I think of flat/neutral as another preference (as is analytical, bassy, V-shaped, etc.).  We shouldn't bias based off of flatness, rather it's ability to reproduce these properties of IEMs (a lot of which is present in an FR graph, but the intricate balance requires more than just a glance to figure how everything will end up interacting). 

 

[Inks]

Quote:

Inks...  The Phonaks (232) have a 8-10dB dip in the midrange (Grey filters...  Get's deeper with others)...  I do have a graph (looks raw) that Phonak provided.  I can't publish it publicly since it was never public material (and it's copyrighted most likely) and I don't have distribution rights (from what I'm aware of). 
   I've seen it's raw graph
 
The rest of the Phonak series isn't as flat either...
 

​

 
There's a nice 4-5 dB V in the midrange with the black filters, even larger with the green (greens veil, blacks as well a little).  The Greys have a nice large spike in the 2.7 area (I know why this is) and have spikes in the rest of the treble (can't explain this one). 
 
  I explained this in my last reply, it's called compensation. Phonak proves to know what they're doing with that response, ~3K region needs the biggest boost to sound flat to the ear. The graph above is pretty much a raw graph, Rin's grey graph below is also a raw graph (notice how they're pretty much the same, though Rin's is more elaborate), it's useless in terms of how it will be perceived, how the average ear will hear it is more important. The results for that is Rin's colored graph. 
 
 

​

 
 
The quality of the 232s beats out that of the 1xx series with grey filters in everything except for clarity...  Flatter isn't always better... This is why I stick with the statement that flatter isn't always better...  It's more than that.  I think of flat/neutral as another preference (as is analytical, bassy, V-shaped, etc.).  We shouldn't bias based off of flatness, rather it's ability to reproduce these properties of IEMs (a lot of which is present in an FR graph, but the intricate balance requires more than just a glance to figure how everything will end up interacting). 
 
  With the first series, the Perfect Fit, Phonak sought out to get the most established method of sound quality while adding their own flavor. It shows, they went for a diffuse-field response but added a bit (emphasis on a bit) of bass and treble, claimed to be preferred by test users. Black filters were added as an option to exuberate those boosts. Well done method, the results and hard work for that sound show, unfortunately the "Perfect Fit" degraded that. 
 
 With the PFE232 I believe it was just a panel of testers, the result was based mostly on preferences of that panel with compromises. Differently development team I believe, different approach. Just by the fact of how they price it, the much superior build and prestige will give you that sense that you're getting something "better". Will need to test that one myself to mention specifics, it's raw graph looks inferior to that of the PFE1s though it may comply better with the forced shallow insertion in the treble, bass seems a tad too boosted. 
 
  Whenever I get the X beats Y in every matter except in A, I will just say preferences. There's nothing tangible there. Neutral can be a said to be just a preference in a purely subjective realm, but it's only common sense to know that getting as close to the source as you can is "sound quality". It's well established in the loudspeaker and source realm. We need to accept that while sound quality may be one matter, there's nothing wrong with liking or preferring what may be rated lower. Again, like I mention, you can have a v-shaped and bassy IEM within the curve. "Analytical" has to be one of the most absurd classifications I have heard, if you call it that....Yes, you need to carefully see every frequency carefully to get an idea of how they interact. 

 

[kiteki]

 
 With the PFE232 I believe it was just a panel of testers, the result was based mostly on preferences [/]
 
  Whenever I get the X beats Y in every matter except in A, I will just say preferences. There's nothing tangible there.

 
In theory in a blind panel test, statistically significant deviance to preference or dispreference is evidence.  However a lot of statistical tests are flawed and the golden ear caveat often messes them up.
 
I think the flat FR advocacy makes sense if the microphones record flat, the speakers play flat, and the room is anechoic.
 
Listening evaluation is quite limited in audio since we've all heard different slices of the cake, this helps the marketing, a lot.
 
There is a lot of anti marketing however a certain amount of it comes from sour apples and narrow thinking.  For example I don't think the countless Clip+ versus HM-801 discussions damaged very much of the sales.

 

[kiteki]

Also I'd like to add I think they're are two kinds of dynamic range, there is very soft to very loud (butterfly to jackhammer) and then there is also very quick volume change and it's precision, like hyper-, selective- or integer- dynamic range.
 
The amount of volume changes and how much within a single second, like a singers vocal quiver, instrument quiver, fast synthetic sounds etc.  Soundstage / imaging / layering / seperation may also help the perceptivity of it.  I think the RE272 and FI-BA-SS have very good hyper dynamic range and it doesn't take a golden ear to hear it.
 
Both of these types which I'll abbreviate as VDR (vast) and HDR (hyper) should definitely be measurable if applied (aside from the soundstage / imaging / layering / seperation precision whatever) since it's only volume fluctuation over time.  I think it relies on factors like the magnet strength and stiffness to weight ratio of the diaphragm, so you can make limited intuitive conclusions from those measurements when they're available.
 
Timbre as I understand it is when you hit middle C on 6 different kinds of pianos and they all produce slightly different overtones.  Ideally an IEM should convey those six kinds of overtones, I think FR extension and linearity helps there, as long as the IEM doesn't layer on it's own special overtone all the time, like a flute would.

 

[tinyman392]

@Inks,
 
Just like you say that people who thought the 232 were technically better is preference, I can also say that the idea that neutral/flat is better is a terms of preference...  There really is no proof that something has to be be flat/neutral to sound the best.  EG, something that is more flat, and more neutral is better... There is no proof the other way around too... 

 

[kiteki]

You can prove which FR is the most popular though - U-shaped.  Just look at all the nightclubs and car audio systems. =]

 

[Inks]

Quote:

Also I'd like to add I think they're are two kinds of dynamic range, there is very soft to very loud (butterfly to jackhammer) and then there is also very quick volume change and it's precision, like hyper-, selective- or integer- dynamic range.
 
The amount of volume changes and how much within a single second, like a singers vocal quiver, instrument quiver, fast synthetic sounds etc.  Soundstage / imaging / layering / seperation may also help the perceptivity of it.  I think the RE272 and FI-BA-SS have very good hyper dynamic range and it doesn't take a golden ear to hear it.
 
Both of these types which I'll abbreviate as VDR (vast) and HDR (hyper) should definitely be measurable if applied (aside from the soundstage / imaging / layering / seperation precision whatever) since it's only volume fluctuation over time.  I think it relies on factors like the magnet strength and stiffness to weight ratio of the diaphragm, so you can make limited intuitive conclusions from those measurements when they're available.
 
Timbre as I understand it is when you hit middle C on 6 different kinds of pianos and they all produce slightly different overtones.  Ideally an IEM should convey those six kinds of overtones, I think FR extension and linearity helps there, as long as the IEM doesn't layer on it's own special overtone all the time, like a flute would.

  That sounds like it's counterpart, signal-to-noise ratio.
 
Yup, good explanation there in regards to timbre, very helpful picture for others to follow. I's the reason why FR response (which includes bandwidth) is very important for timbre, an even frequency response will allow all those overtones to have an even distribution. 
 
Quote:

@Inks,
 
Just like you say that people who thought the 232 were technically better is preference, I can also say that the idea that neutral/flat is better is a terms of preference...  There really is no proof that something has to be be flat/neutral to sound the best.  EG, something that is more flat, and more neutral is better... There is no proof the other way around too... 

  Perhaps liking FLAC in comparison 128kps is also subjective (given the differences are there), speaking in those terms. Sonic purity, is sonic transparency allowed by an even FR.  It doesn't have to be completely neutral to sound best, it's a big part of the recipe though, that's what we try to achieve with transparent sources and high-res files. I believe we are highly biased by price and group reaffirmation, neutrality is looked for in the highest end of loudspeakers. There is also proof that even those without knowledge, recognize neutrality as having more fidelity, perhaps an innocent approach to the whole thing enables a more honest evaluation for those users.
 
To consider there are also products that while being neutral in FR, may do poorly in the other tests I evaluate, looking into decay/transients (shown on CSD), bandwidth, dynamic range and soundstage presentation. The Etymotic MC series comes to mind, it seems fairly below average in most accounts except FR distribution....

 

[tinyman392]

Quote:

  That sounds like it's counterpart, signal-to-noise ratio.
 
Yup, good explanation there in regards to timbre, very helpful picture for others to follow. I's the reason why FR response (which includes bandwidth) is very important for timbre, an even frequency response will allow all those overtones to have an even distribution. 
 
  Perhaps liking FLAC in comparison 128kps is also subjective (given the differences are there), speaking in those terms. Sonic purity, is sonic transparency allowed by an even FR.  It doesn't have to be completely neutral to sound best, it's a big part of the recipe though, that's what we try to achieve with transparent sources and high-res files. I believe we are highly biased by price and group reaffirmation, neutrality is looked for in the highest end of loudspeakers. There is also proof that even those without knowledge, recognize neutrality as having more fidelity, perhaps an innocent approach to the whole thing enables a more honest evaluation for those users.
 
To consider there are also products that while being neutral in FR, may do poorly in the other tests I evaluate, looking into decay/transients (shown on CSD), bandwidth, dynamic range and soundstage presentation. The Etymotic MC series comes to mind, it seems fairly below average in most accounts except FR distribution....

 
You can prove that FLAC is a better source than 128 kb/s due to some reasoning, 128 loses out on some bits (lots of bits actually).  However, with FR, it doesn't completely show that you lose out on anything if it's still there.  If neutral FR was the target for ultimate fidelity our search would've been over eons ago.  Compare a neutral RE0 to a PFE232, don't deny the 232 is better technically over the RE0 now despite the fact that the RE0 is a lot more neutral than the 232.  The thing about proving something wrong is that it's a lot easier than proving something right as you only need one true counter-example.  This being one of the many.  
 
I'll agree that flat response generally will be better, but won't always be better.  A pattern on Head-FI that I see is that if they see an FR graph that is bass heavy, or shows a V in it they will automatically dismiss it as being poor quality.  When in fact it isn't.  I don't want this idealism to stick around.  I honestly think it goes much deeper than just FR.  

 

[eke2k6]

Ok, let's leave Inks in peace to do what he intended with his thread. As much as we disagree, he still has the right to have himself heard.

 

[Inks]

Ok, let's leave Inks in peace to do what he intended with his thread. As much as we disagree, he still has the right to have himself heard.

  The discussions have been fruitful for the most part, like I mentioned in my first post, they are welcomed. 
 
Quote:

You can prove that FLAC is a better source than 128 kb/s due to some reasoning, 128 loses out on some bits (lots of bits actually).  However, with FR, it doesn't completely show that you lose out on anything if it's still there.  If neutral FR was the target for ultimate fidelity our search would've been over eons ago.  Compare a neutral RE0 to a PFE232, don't deny the 232 is better technically over the RE0 now despite the fact that the RE0 is a lot more neutral than the 232.  The thing about proving something wrong is that it's a lot easier than proving something right as you only need one true counter-example.  This being one of the many.  
 
I'll agree that flat response generally will be better, but won't always be better.  A pattern on Head-FI that I see is that if they see an FR graph that is bass heavy, or shows a V in it they will automatically dismiss it as being poor quality.  When in fact it isn't.  I don't want this idealism to stick around.  I honestly think it goes much deeper than just FR.  

  128 kb/s is inferior because it makes the frequencies of the recording peakier, FLAC is closer to flat, that's a big chunk of what makes it better. In the same way an IEM drawing further away from flat, is taking you even further away from the original purity of the recording, preferences aside. I would have to test both the RE0 and PFE232 with my approach, the RE0 may be closer to flat, but it has high consistency issues IME, peaky/grainy and not well matched. I don't look into price or how a manufacturer places a certain product in their line-up, I look solely into performance. Do note this though.
 
Westone's most flat IEM is the Westone 4. Westone's most flat custom is the ES5
Hifiman's most flat IEM is the RE272 
Brainwavz/Fischer's most flat IEM is the B2/DBA02
Grado's most flat IEM is the GR10
Sony's most flat IEM is the EX1000
Earsonic's most flat IEM is the SM3
Vsonic's most flat IEM is the GR01
AKG'st most flat IEM is the K3003
Jays' most flat IEM is the Q-Jays
Phiaton's most flat IEM is the PS200
 
These are all their current flagships!
 
Big exceptions are Shure, Sennheisser and UE, but their most flat are the next ones up (SE425/UE700/IE7). And in these cases, the flagships are fairly close behind these and flatter than the others in the manufacturer's line-up.