[davvy]

Just came across this article on another forum and thought some of you might be interested.

http://www.geocities.com/altbinaries...l/mp3test.html

I would have done the test slightly differently myself, but it is by far the best attempt at a proper scientific blind trial of MP3 vs WAV on high end gear that I have seen.

To me the results basically lend support to the beliefs of those of us (myself included) who believe that high bitrate MP3s are basically indistinguishable from the original CD.

Unfortunately the authors don't state which encoder they used, but it is also likely that their 256k MP3s were in fact inferior to Lame encoded MP3s using the -alt presets (and still they couldn't tell a difference).

Anyway, interesting to hear some opinions on this.

 

[sleepkyng]

320kbps mp3 dont' match up to cd quality sound imo
but i think that has to do with the source,

my powerbook/modded sonica source doesn't really compare to my nec cdr/modded art dio combo (even tho when i first got both of these setups, i couldn't really distinguish which was better)

now i can really tell, partly because my ears are better at picking up subtle nuances.

 

[Edwood]

Unless you are tone deaf, it is very easy to tell the difference on high end equipment.

-Ed

 

[dd3mon]

Quote:

Originally posted by Edwood Unless you are tone deaf, it is very easy to tell the difference on high end equipment. -Ed

Agreed - my soundcards can't show me though. If I a burn two wavs, one the original, and one that has been from wav -> mp3 -> wav (identical to mp3, just in wav format so I can burn it as cdaudio track), then play them on my cd player the difference is clear. High frequencies are the downfall of mp3. Midrange is decent, lows aren't too bad. Overall some detail is lost, but the high frequencies really stick out to me.

This is comparing Lame vbr (-preset standard) to original wav.

-dd3mon

 

[davvy]

Quote:

Unless you are tone deaf, it is very easy to tell the difference on high end equipment.

Have you actually read the article I posted the link to ?

The people in the test were clearly not tone deaf, and the equipment was about as high end as it gets. They didn't find it easy to tell the difference.

Please don't bother posting in this thread if all you have to say is something along the lines of "I can clearly hear a difference" or "MP3s are crap", unless you have some sort of evidence to back up your statements.

 

[Iron_Dreamer]

Quote:

Originally posted by davvy Have you actually read the article I posted the link to ? The people in the test were clearly not tone deaf, and the equipment was about as high end as it gets. They didn't find it easy to tell the difference. Please don't bother posting in this thread if all you have to say is something along the lines of "I can clearly hear a difference" or "MP3s are crap", unless you have some sort of evidence to back up your statements.

Maybe you have spent to much time at hydrogenaudio. I just reencoded all the files for my Archos from lossless archives to LAME 3.96 Preset Standard, and it is not too hard to tell the difference on my system, especially with anything complex and layered, the soundstage really starts getting compressed (And yes I used the foobar ABX just for the hell of it to confim my findings).

 

[Wodgy]

Davvy, this study proves the exact opposite of what you say it does. They found that, even at 256kbps, there were musical pieces where the participants consistently found differences between the CD versions and the MP3s; the only oddity is that in those instances people seemed to usually prefer the MP3s. The key thing is that there were musical selections where people could consistently tell a difference.

I personally don't find the fact that they often preferred the MP3s all that odd either. MP3s tend to have a slightly warmer sound (this is especially clear at lower bitrates, but I'd imagine that it extends consistently, though less obviously, to 256kbps), and a warmer sound is more like real music and less "hifi".

 

[davvy]

Quote:

They found that, even at 256kbps, there were musical pieces where the participants consistently found differences between the CD versions and the MP3s

No they didn't. Some of their statements are not well put. Prehaps a little was lost in translation. However, I have re-read the article carefully and nowhere do they say this.

They do say that there were some samples where 128k MP3s were preferred over WAVs by quite a few people: Quote:

The fact that some of the 128 kbps samples were consistently judged to be better than their original CD counterparts by this skilled group – even by the best among them

Take a look at the data table they present at the bottom of the page. There is no track where people consistently were able to pick the MP3s. And there is no one person who was able to get it right with anything approaching consistently. On the 2 "best" tracks, 9 people out of 14 got it right, but this is probably just random variation. There are also 2 tracks where 10 out of 14 people scored zero.

Some of what is in the article can get complicated and somewhat confusing, but their conclusions are clear: Quote:

In plain language, this means that our musically trained test listeners could reliably distinguish the poorer quality MP3s at 128 kbps quite accurately from either of the other higher-quality samples. But when deciding between 256 kbps encoded MP3s and the original CD, no difference could be determined, on average, for all the pieces. The testers took the 256 kbps samples for the CD just as often as they took the original CD samples themselves.

 

[Wodgy]

The data they present in the table isn't rich enough to make any real inferences. The problem is their filtered scoring system, which distorts the 256kbps results by lumping them in with the 128kbps results. I don't think anyone is truly interested in the 128kbps results; it's the 256kbps results that matter, and their scoring system does not allow for proper conclusions, e.g.:
"But no points were awarded at all if the 256 kbps sample was correctly identified but the 128 kbps and original CD samples were reversed."
No points for making a correct identification? Ludicrous.

Because the data they present is inadequate, we're forced to go by the article text. And it seems to be clear from the text that there were pieces where, even at 256kbps, differences were apparent. They state:
"For those pieces which our listeners most frequently guessed wrong, the MP3 encoded samples were judged in general to be superior to the CD sample."
This appears to imply that there were pieces that listeners frequently guessed wrong. However, in the next paragraph they aggregate all the data together and it becomes worthless except for one specific conclusion that they draw: on average, across all people and musical pieces, there was no statistically reliable difference between 256kbps and CD. But this doesn't allow us to conclude that there were not specific pieces where people frequently guessed wrong, and their earlier text encourages us to conclude that.

That said, we don't need to parse the article text deeply. The very fact that their test had "winners" who scored better than average (despite the wonky scoring system) proves that some people have an ability to tell the difference between MP3s and CDs. People on average may not, but for those people who can, they seem to have an ability to do it on a semi-consistent basis.

 

[Earwax]

Quote:

Originally posted by Wodgy That said, we don't need to parse the article text deeply. The very fact that their test had "winners" who scored better than average (despite the wonky scoring system) proves that some people have an ability to tell the difference between MP3s and CDs. People on average may not, but for those people who can, they seem to have an ability to do it on a semi-consistent basis.

I agree with your conclusion, but remember this is a four year old article with 256k CBR mp3s. It says nothing about whether or not a similar conclusion would be reached if the best encoder we have today were used instead.

 

[sleepkyng]

if it's not lossless, then how could one make an argument that it would be as good?

lossless= no loss of sound quality

nonlossless=loss of sound quality

well, regardless, to my ears, there is a difference, but again, maybe it is the sources...

 

[pbirkett]

I agree that people can tell the difference (even blind tested).

What I disagree with, is the people who say the differences are MASSIVE. When they clearly arent. People have a habit of exagerrating what they say they can hear. Even if some people say they heard a difference, they will automatically say its a massive difference. Its just the way audiophiles are.

Really though, who cares? The vast majority of my MP3's are certainly listenable, and I dont sit there picking faults with them, like "ooh the cymbal sounds slightly too metallic", or whatever. I encode all my music in MPC format, so really, I couldnt give a flying f.........

 

[bangraman]

...ig leaf. I totally agree.

 

[minya]

I'm with pbirkett. The differences are there, but they're not as great and enormous as the length of the Grand Canyon. Of course, those differences matter to me, so I greatly prefer lossless formats. But MP3 performs within its niche (small, easily transportable) exceedingly well, so I'm very fond of it.

- Chris

 

[davvy]

Quote:

I don't think anyone is truly interested in the 128kbps results; it's the 256kbps results that matter, and their scoring system does not allow for proper conclusions

This is very true. It is a pity (this being the only properly conducted blind test on high-end equipment that I have found) that they didn't conduct their test differently, particularly with regards to their scoring system. However, I do think it still allows us to draw some limited conclusions and one of those is that telling high bitrate MP3s from WAVs is at least very difficult even on high end equipment.

Quote:

That said, we don't need to parse the article text deeply. The very fact that their test had "winners" who scored better than average (despite the wonky scoring system) proves that some people have an ability to tell the difference between MP3s and CDs. People on average may not, but for those people who can, they seem to have an ability to do it on a semi-consistent basis.

This is true for 128k MP3s, but I don't believe you can say that has been proven in this article for 256k MP3s.

Quote:

remember this is a four year old article with 256k CBR mp3s. It says nothing about whether or not a similar conclusion would be reached if the best encoder we have today were used instead

If only we could get them to repeat the test, but only compare -alt preset LAME MP3s with WAVs.

Quote:

What I disagree with, is the people who say the differences are MASSIVE. When they clearly arent. People have a habit of exagerrating what they say they can hear. Even if some people say they heard a difference, they will automatically say its a massive difference. Its just the way audiophiles are.

I guess I pretty much agree with you. There is no doubt that there are differences - over at hydrogen audio there is a list of samples on which there is a difference. I just believe that the differences are so small that the majority of the time they are not detectable even by trained listeners on high end equipment.