[Piotr Michalak]

OMG. Never was my intention to produce such a shitstorm. I just answered Meshails question for a recommendation of a neutral top-tier IEM... And I said I applaud Crin for all he did. And I said I am not satisfied with a ranking (the ranking?). And I was even being called a spammer for answering the aforementioned question with a link to another forum, a normal practice within head-fi. What a strange place this has become! Better stick to classifieds.

 

[bamboostreet]

Dude just don't base your purchase decisions on one single person or list.

They all have the same list! When you get free samples, this is exactly what happens: I look at 5 different lists and everyone, ALL of them, they all made the same list, hexa, simgot, tangzu, s12, cadenza, etc, etc, etc...

 

[goodvibes]

Here's the thing: I do. Constantly.

As much as I can, whenever I can, possibly to the annoyance of my audience. I can't link anything here due to MoT status but if you do a quick search, you'd find that I had dedicated an ENTIRE 12 minute video to speak on the things that affect the objectivity of my reviews, the conflicts of interest, and in general just the reasons why you SHOULD NOT trust me. This videos is linked back to on many videos thereafter and emphasised on the video itself that one should watch it before coming to any hard conclusions.

My latest video, talking about the new Ranking List, is a prime example where I constantly emphasise not to take me as gospel amidst trying to explain the changes and the data analysis of it all. It's a fine balance between having to do disclaimers and putting out the actual content, and spending 5+ minutes telling people about disclaimers in a 10 minute video is not great for audience retention.

Barring the use of a pop up disclaimer page that activates when someone visits my ranking list, I'm not sure what else I can do. You call it a "cozy spot to hide behind", I call it the wall of text that nobody wants to read, no matter how hard I try.

But beyond that, there is a simple explanation to why you don't seem to agree with me "these days".

At the beginning, everyone is new. They see a new Ranking List and simply do not have the experience to refute it. I was like that at some point, so were you. You had not established the parameters in which you found something "good" or "bad", and therefore had no choice but to take the list at face value.

As you grow within the hobby, finding all of that out on your own, your own tastes and personal indicators of performance, you'd naturally diverge from everybody else. Not just me, not just your friends, EVERYONE. You as a person began understanding yourself more within the context of the hobby, forming your own opinions and biases, and I think that's beautiful. No two people would ever share the same set of opinions on a macro scale, and if we sat everyone down and had them do a ranking list themselves I am 100% confident we will never ever find a duplicate.

So no, the Ranking List didn't get worse. I was never a community consensus. You simply grew and found your own answers, as you should.

So this and I say that while wishing there were more things included in the measurements like impulse and phase and while I respect what you're doing as much as anyone, I don't always agree with the rank which is how it should be. We all have different parameters and I so appreciate that you're attempting to get beyond them with a more analytical take even though it's obviously never going to be perfect. The takes are also somewhat limited and purposely so. You can't go through every iteration of cable, tip or source interface on every device you measure and unfortunately, that can radically change an 'opinion' on a piece (if it's close enough to respond). I appreciate that you emphasize the technical aspects to try and keep it real. We're human so can't avoid opinion but here, it is much more tempered than most.

For instance your take had an influence on my beloved S8 and prefer it to many TOTL I've heard but without Comply's and the right silver cable, I'd have moved on. Its great info retrieval and special solidity/cohesiveness made the effort of taming it's pinna gain worthwhile. I have since tried others at much higher prices and configs but, for me and while some of those others may do certain things more interestingly, they never sound quite as right or cohesive as configured. I have short periods of enjoying something different but... What's interesting is that I'd like to get customs again but I'd never send the S8s in for that without some ability to tame them with tips.

Much of this is interface so it's easy to understand such varied opinion on things. There's a reason there is much more disagreement than agreement with IEMs and people that swear by their recent device eventually move on to their next nirvana but... If something was ever actually great, it will continue to be so forever.

 

[goodvibes]

OMG. Never was my intention to produce such a shitstorm. I just answered Meshails question for a recommendation of a neutral top-tier IEM... And I said I applaud Crin for all he did. And I said I am not satisfied with a ranking (the ranking?). And I was even being called a spammer for answering the aforementioned question with a link to another forum, a normal practice within head-fi. What a strange place this has become! Better stick to classifieds.

Well, you probably didn't intend for that to sound as it did. I do find it interesting that you recommended two Polish devices and how that corresponds to your location. We're all influenced by our biases whether apparent or not and I'm sure Crin is not immune even though he goes as far as anyone to try and remove that.

I appreciate that Crin is willing to mix it up on his list as his perspective may change over time. Obviously nothing radical happened so no big deal but you made it appear so.

The bigger issue is that readers take it as gospel as opposed to a guide/recommendation with the knowledge that YMMV.

Look at the IEMs I like. We have a couple very respected reviewers that have them near the top of lists and other than really don't care for them and I can see both sides of that.

 

[goodvibes]

They all have the same list! When you get free samples, this is exactly what happens: I look at 5 different lists and everyone, ALL of them, they all made the same list, hexa, simgot, tangzu, s12, cadenza, etc, etc, etc...

Not all and for some, interest and controversy is clicks so maybe you're following the wrong guys. I do think many reviewers are too caught up in response as being all important when it's only part of the recipe because it's something easily displayed. I'm glad Crin does comment on more than that even though he was a relative pioneer in comparative response.

 

[Piotr Michalak]

Polish devices

Ambient Acoustics is from Ukraine. Never even been there. I owned CIEMs made by three other Polish companies apart from Craft Ears and didn’t recommend them. I appreciate your comment but wanted to get this straight.

 

[antdroid]

I don't know the exact methodology that Crin published on his latest ranking but he did mention several times that he is curving it on a bell curve. For my personal ranking system, I also use a Normal Distribution to adjust the curve using statistical formulas in Excel. What does this mean? It means not everything can be S and not everything can be F. It means the distribution is spread through out the grades with the majority falling between D and B.

Here's a copy/paste of a typical normal distribution curve (pictured below). 68% of the population will fall between -1 and 1, (i.e. D and B, with C being 0). 95% will fall between -2 and 2 (E and A), which means only the top 5 and lower 5% are going to be in the higher echelon, and the top and bottom 1% are below -3 and above 3 (F and S).

edit: highlighted personal just to emphasize, that its my personal ranking list. My scoring is also weighted with preference and tonality heavier than technicality (all of which is subjective)

 

[Andrew_WOT]

 

[emdeevee]

I don't know the exact methodology that Crin published on his latest ranking but he did mention several times that he is curving it on a bell curve. For my personal ranking system, I also use a Normal Distribution to adjust the curve using statistical formulas in Excel. What does this mean? It means not everything can be S and not everything can be F. It means the distribution is spread through out the grades with the majority falling between D and B.

Here's a copy/paste of a typical normal distribution curve (pictured below). 68% of the population will fall between -1 and 1, (i.e. D and B, with C being 0). 95% will fall between -2 and 2 (E and A), which means only the top 5 and lower 5% are going to be in the higher echelon, and the top and bottom 1% are below -3 and above 3 (F and S).



edit: highlighted personal just to emphasize, that its my personal ranking list. My scoring is also weighted with preference and tonality heavier than technicality (all of which is subjective)

If I understand this correctly, curves like that introduce a very strange element to the situation, potentially changing a prior grade purely for statistics sake without regard to anything else.

 

[Edric Li]

Crinacle currently ranks about 2000 IEMs. Assuming that he has all 2000 IEMs in front of him, that he can tell which IEM is better than another based on a pairwise comparison, and that he uses a theoretically optimal sorting algorithm, it will take him log_2 (2000!) = 19053 pairwise comparisons to produce his ranking list. If he takes 2 minutes to compare 2 IEMs, spending 1 minute on each, it will take him 19053*2/60/24 = 26.5 days to produce his ranking list.

More realistically, assuming that he reviews IEMs in batches of 10, for a total of 200 batches, and that he has to tell you the ranking of the IEMs from the batches that he has gotten so far, It will take him log_2 (10!) = 22 pairwise comparisons to rank each batch, and 10+20+...+1990+2000 = 20100 pairwise comparisons to merge the ranking of the new batch into the bigger list, for a total of 22*200+20100 = 24500 pairwise comparisons. If he takes 2 minutes to compare 2 IEMs, that will be 34 days.

 

[Andrew_WOT]

The whole IEM market is over bloated, 2000 models , that never happened with full size headphones. We had what, at most a dozen models worthy ranking.

 

[antdroid]

If I understand this correctly, curves like that introduce a very strange element to the situation, potentially changing a prior grade purely for statistics sake without regard to anything else.

It can potentially, but things naturally fall into some sort of normal distribution when there is a large enough sample population -- hence why there are statistical analysis around this. That said, it's just a way to normalize and rank things -- since people like rankings. (For reference, I do update mine from time to time, but inputting raw values, and letting the math do its work updates the rankings for me) -- the data on my site shows both raw score and normalized score. This method also allows the data to skew slightly based on new technology or tuning changes with less work. What was considered above average yesterday could be average/mediocre today or tomorrow.

Anyway, I don't want to steer too far away from Crinacle's thread here. I'm just saying at the end of the day, its a subjective task of giving scores, rankings, reviews, etc. What I like is not what Crin may like and may not be what user X may like and may not be what you like. A lot of the new reviewer generation here provide our own measured data, with target curves, and descriptive reviews to help users understand what WE like and what WE think is considered neutral. But neutral is a debatable term, and what music I listen to is not the same as what you may listen to and all of this gives into all of our collective biases and firsthand impressions.

Don't take these rankings too seriously. Its just a data point. We are blessed today to have so much data available to us on the internet, and users should use that to make their own choices, and not rely solely on one data point and expect it to match with their preference.

 

[goodvibes]

Ambient Acoustics is from Ukraine. Never even been there. I owned CIEMs made by three other Polish companies apart from Craft Ears and didn’t recommend them. I appreciate your comment but wanted to get this straight.

I don't think he'd like the response measurement of that one.

 

[1TrickPony]

Damn! Never seen Crin this angry. I agree everyone's reality/preference/cup of tea is different and we should never impose our likes/dislikes on others. Everyone hears differently and we must respect that.

He's basically become a cult figure, with such influence comes great responsibility. Also, one does go to Mordor and takes on Queen Bey.

 

[FcConstruct]

Crinacle currently ranks about 2000 IEMs. Assuming that he has all 2000 IEMs in front of him, that he can tell which IEM is better than another based on a pairwise comparison, and that he uses a theoretically optimal sorting algorithm, it will take him log_2 (2000!) = 19053 pairwise comparisons to produce his ranking list. If he takes 2 minutes to compare 2 IEMs, spending 1 minute on each, it will take him 19053*2/60/24 = 26.5 days to produce his ranking list.

More realistically, assuming that he reviews IEMs in batches of 10, for a total of 200 batches, and that he has to tell you the ranking of the IEMs from the batches that he has gotten so far, It will take him log_2 (10!) = 22 pairwise comparisons to rank each batch, and 10+20+...+1990+2000 = 20100 pairwise comparisons to merge the ranking of the new batch into the bigger list, for a total of 22*200+20100 = 24500 pairwise comparisons. If he takes 2 minutes to compare 2 IEMs, that will be 34 days.

I'm loving this math. Just imagine Crin standing in the middle of a sea of IEMs and cables with nothing but a laptop and a mug of coffee ranking and comparing IEMs for a month straight.