Very interesting article by Tyll with regards to headphone burn in - Page 3

Yes.

And let's take this a couple steps further. Suppose burn-in is real. Manufacturers could then start selling "factory certified burned-in" pairs for a premium. Why don't they?

Also, I find it curious that manufacturers don't publish their data on burn-in. Most pairs go through a few years of R&D before release. Didn't Sennheiser put about six years into the HD-800? One would think the drivers have been run for tens of thousands of hours in testing and measured quite carefully. If burn-in exists, then it would surely turn up during R&D. If so, then I'd imagine the manufacturer would either include burn-in instructions with the headphones or burn-in at the factory.

Beancounters and lawyers don't like products that change during the warranty period. That opens up liability and potential returns - those are very expensive. I can't imagine any manager approving the shipment of a product that would substantially change. That's just too risky for a business.

I want to congratulate Tyll for conducting a nice piece of suggestive work. It doesn't qualify as a scientific study in its present form, but provides a fine basis for one.

 

Having just skimmed the article itself, one or two changes occur to me off the top of my head: (1) a pre-test (blind) - no music playing - to see whether Tyll and his colleague could distinguish which pair of headphones were on their head over a fixed number of trials. This would control for "head feel" as the source of the ability to tell them apart; (2) the second set of trials was terminated at 15 of the intended 30, because 13 had been 'identified' (or it could be 'guessed') correctly. That's a shame. This involved a binomial test, akin to flipping a coin. Sometimes, a fair coin does turn up 13 heads (or tails) in 15 tosses. There's absolutely no law to prevent this. In a statistically-based study, all intended trials should be completed.

 

Notwithstanding the last point, an interesting observation in Tyll's report is that he could [words to this effect, not a direct quote] "easily tell which headphone he was listening to" in the second set of trials. This highlights the difference between effect-size - the magnitude of a difference - and 'statistical significance'. The latter concept - which evaluates how often chance would produce the same result - is not nearly as important as one or two other posters seem to have imagined, although it still has its place. The former concept is something which statistically-based research is increasingly required to report if it is to be considered for publication.

 

To give a (not very good) example of effect-size, it doesn't take a statistically-significant number of repeated trials to disclose that (track sprinter) Usain Bolt won his 100m/200m gold medals by significant margins. It's the margin that's important, not the question of chance.

 

Tyll seems to be reporting having acquired a similar performance level with respect to distinguishing these two 'phones, whatever the salient factors may have been. His further and more carefully controlled work - or even better, a replication by someone else as Tyll has fully disclosed his methodology - could go a long way to isolating these salient factors.

 

Again, good work Tyll.

All that article did is make me want to sell my K701s and get the pretty Q701 White/Black model.

I'm not trying to accuse Tyll of cheating, but his laptop screen did have a glossy surface. It is possible, I'm not saying he did, but it's possible that he could have seen the headphones' color from the reflection in the screen.

It's very possible, that's why I tipped it back quite far so that I couldn't see. It really boils down to the difference between the two headphones being quite easy to hear.

Hi Tyll, this seems the decisive point. If we start to think that head feel or subliminal perception of a reflection in a screen (which you have ruled out) affects what we hear to the easily apparent extent ("quite easy") you noted - well, that would be an interesting finding of another sort! The sort that falls into the placebo/cognitive bias camp. It would suggest the effect of placebo is quite large - but there would need to be a lot of very careful work to quantify this under various conditions before it could be supported as a genuine factor...
 

The other line to investigate is that this is exactly what it seems to be - a real difference in the sound of the two 'phones. IMO anetode's point (9/9/11) needs to be taken into account in future tests - how different did they sound before burn-in? Assuming they didn't sound different, or not as different as after burn-in, and the finding replicates with more samples (believe me, they often don't - one of the perennial frustrations of doing science!) then you have a solid finding here.

 

Very interesting. Cheers! Andre

 

 

+1

 

 

While Tyll is describing a difference, it's only a difference. I don't believe he ascribed a value to it, good, bad or preferred. Maybe some might like it one way, someone else might like it as "new." That it exists is neither good or bad, only that something changed, is observable, and is measurable.  

 

 

There's not necessarily any observed changes, because the headphones weren't ABX tested before burn-in.

 

There seem to be measurable differences, yes, according to Tyll's older graphs.

^ True. (Edit: sorry Head Injury - this post based like yours on Rdr.Seraphim's)

 

The observable, measurable bit takes care to quantify however. Indeed, IIRC Tyll did not qualitatively characterize it as good, bad etc; he also did not set out a scale of differences so that other observers could - all else being equal - be trained to hear what he heard. IMO, such a scale is needed as one of the next steps.

 

His report provides some clues how to do this. He used the term "quite easy", which of course has a relative meaning ("hard to tell" < "quite easy" < "easy" < "very easy" for example). More valuably, he also supplied some descriptive terms (IIRC "smoother" was one of them), which is what is needed before other listeners can be trained to repeat this sort of experiment. To construct a scale one would need to link these and other terms together; off the top of my head, one such scale-type that might be appropriate here is semantic differential.

 

This sort of scale doesn't imply value-judgments like good or bad. Its function is: first, to make the listening trainable; two, to assign numbers or some similar token to the scale to assess how different (*); three, to assess (during training) that measurement can be achieved repeatably/reliably between observers (inter-rater reliability or inter-observer agreement are the terms used). Assuming adequate reliability is achieved - this btw would be an achievement in itself - then Tyll's experiment can be expanded into a larger and better study.

 

On another point, imagine if headphone burn-in was confirmed, and it turned out to be the sound of the first 100 hours one preferred... 

 

Note: (*) as written this implies a one-dimensional or single-factor scale. It's more likely to be multi-factor., i.e. this 'phone after burn-in had more x, less y, same z etc. This is just as easy to handle but does need to be handled - a one-dimensional scale for a multi-dimensional phenomenon gives us noisy and misleading data, obscuring real differences or creating false ones. Construction of good measuring instruments is absolutely essential to conduct science. It takes time.

Edited by AiDee - 10/21/11 at 12:52pm

Doesn't that sort of negate everything?

It leaves the possibility of product variation wide open. That's been discussed a lot in this thread.

So you're saying that the bottom line is:  Tyll has two 702 headphones that don't seem to sound exactly the same and any further conclusions are speculation ?

 

Basically.

 

Again, there are measurements which suggest burn-in exists, but without at least trying to account for product variation, we still don't know how audible it is.

 

To really be sure, Tyll would have to find two separate K702s that sound so similar he can't ABX them before burn-in, then see if he can ABX them after he burns one in. If product variation is large, that might get expensive 

Did Tyll make frequency plots of the headphones before starting?  If so, how similar were they?