BLIND TEST: Lossless vs. MP3 320 - Page 5

Quote:

Originally Posted by DeadBeats 

What about printscreens ?

of what? the ABX window itself before you click exit? I'm no photoshop expert but i know it's not hard. the colors surrounding the square are very very easy to fake and copy. substiuting anything there would be a piece of cake for many. plus a print screen of that is suspiciious. what about the text editor after you click exit? ...even more easy to fake...you can just edit it then..in the text edit itself and dont' need to even use photoshop or something

You technically don't need to photoshop - you could replace file "B" with a completely different song, so you'd never get the test wrong.  the resulting window would show your perfect results, which would in turn show on the screenshot. 

Still, I accept some people who need their ego stroked may come into these threads posting, "X is clearly better than Y, check the bass!" and leave.  I cannot accept the idea of someone taking the time to falsify ABX tests, solely to bolster their false claims.  No one could possibly be that pitiful - and if they are, I say we let them continue with the charade, because their self esteem must be INCREDIBLY low. 

Here's the funny thing: Most people wouldn't have even thought to cheat until Jaywalk3r brought it up.

Jaywalk3r, since you seem to like to crap on blind tests, can you design for us a foolproof one so we can actually gets something done that you'll approve of?

hmm my experience as a "netizen" tells me no need to doubt ppl on such harmless topic...

btw I think practice makes perfect, sites like mp3ornot or klippel.de are good place to start

Well speaking for myself - I prefer people posted the logs.  I'd like to believe MOST people will have done it legitimately.

Of course feel free to keep scoffing at the legitimacy of the results 'if you must'.  I for one don't really care.

@Jaywalk3r

Are you suggesting that this track (or 320mp3) is impossible to ABX? So any positive result must be due to a dirty lying scumbag?

Because that would be treason against the very concept of lossy encoding itself.

Anyway, I can see it's just a misunderstanding, but wouldn't you agree that an opinion posted with an ABX log carries more weight than a one liner?

A person who even considers faking the log must have at least done the ABX properly a few times, and if that person is reasonable, they'd also jump on the 'failing bandwagon'. Because, there's nothing to lose, and lets face it, you're supposed to fail.

Quote:

Originally Posted by bowei006 

see :D let's all be good head fi samaritans here :D we shold have a mass blind test thread..but with many different music genres like 3 songs in each genre. and then average the results and see which genre is easier to tell apart from lossless and lossy. im voting on..........modern rock/metal

I tried to find a correlation between genre and the ability to be able to successfully ABX lossy, and I've failed thus far.

I'd also think that modern rock/metal should be easier, but it doesn't seem so from my limited testing, because with a lot going on, it makes it even more difficult to pick out the artifacts. On the other hand, some simpler tracks seem downright impossible because the encoder has more bits to throw at whatever's there.

The only thing I've found is that tracks with very sharp transients or lots of electronic sound effects seem to be the easiest. Killer tracks, as they say.

And in my experience, loud listening is absolutely critical (I usually ABX at 90db with my super accurate pain-o-meter), as is proper selection of a slice from the track to test (few seconds max + small soft lead in), because sudden transients can appear to sound very different due to yo' ears trying to compensate/save themselves.

kn19h7 said it best, practice makes perfect, because we are automatically accustomed to listening for differences in 'tone' in our day to day lives, and not hunting for barely audible artifacts/differences in attack.

All for naught it would seem, because even with the tracks I successfully ABX, the differences practcally vanish under normal listening.

I shall try this now!

PS. Found this explaining the ABX thingy:

http://www.youtube.com/watch?v=jt7GyFW4hOI

All I can say is total torture, I count that as a fail really. The clicking instrument seems to hit with a slightly different tonality in A vs B. Otherwise differences are so so subtle that I really can't tell which one is which, I had to turn the treble on my monitors ALL the way up to hear that tiny tiny difference. If I have to pick, maybe A? Voted no diff anyway.

Quote:

Originally Posted by Jaywalk3r 


I in no way "crapped on blind tests." I'm a fan of properly performed blind tests. I just think it's absurd to only believe someone's test results if they post an easily faked log. It's like a corporation's human resources requiring an ID to verify identity for employment processing, but allowing handwritten "Hi, my name is …" name tags to meet that ID requirement. It makes the requirement meaningless.
If the standard of "proof" is so low, there's no reason we shouldn't simply take someone's word that they ABXed two samples and were able to tell the apart 18/20 times, p-value 0.02%. The attitude that "your comments aren't worth a hill of beans in this thread unless you post logs" is pointless. Logs or not, we're taking posters at their word. There's nothing wrong with that in an informal setting such as this, but it certainly isn't a foolproof system by any stretch of the imagination. These blind test threads, while fun, are not rigorous scientific experiments. The results must be taken with a grain of salt.
If one wants to trust the results of blind tests for a group of people, those being tested should not have access to the results until after they have been formally recorded. The testing process and experimental conditions need to be much tightly controlled. There is a reason authors of studies published in peer refereed scientific journals rarely, if ever, reference Internet forums such as these as sources of credible data. If you want foolproof, the ability to withstand peer review before and after publication is critical.
On the other hand, if you just want entertainment, do some Foobar ABX testing and share with us the results. If you want, you can even post the log. Or not. It doesn't really matter.

No it's not.  Most people in the Sound Science forum are here because it is a respite from the rest of head-fi where 'differences' are waxed on about lyrically for paragraph after paragraph.  We choose to believe that most people here will choose to do the test honestly if they are going to contribute to the conversation and actually care about educating themselves.  The standard that we have chosen to set here is that if you are going to comment, post your logs.  Here's the thing about your attitude - it doesn't matter if some people fake their results.  If we choose to continue these kind of tests here, eventually people will know who they can trust and who they can't, and will make up their own mind whether to believe them based on how they compose themselves here over time.  At least here we have some hurdle in place to take part in the conversation, as opposed to the main equipment forums where suggestions are given every day by people who haven't heard the gear, or who have made only sighted listening tests, often in a noisy meet environment, and where the most respected reviewers are the ones who write the most extensive positive pieces about advertisers gear, which, surprise surprise, gets them asked to do more reviews. 

why is it in every second thread i post in, there are people insulting each other...why?