[bowei006]

Quote:

What about printscreens ?

of what? the ABX window itself before you click exit? I'm no photoshop expert but i know it's not hard. the colors surrounding the square are very very easy to fake and copy. substiuting anything there would be a piece of cake for many. plus a print screen of that is suspiciious. what about the text editor after you click exit? ...even more easy to fake...you can just edit it then..in the text edit itself and dont' need to even use photoshop or something

 

[El_Doug]

You technically don't need to photoshop - you could replace file "B" with a completely different song, so you'd never get the test wrong.  the resulting window would show your perfect results, which would in turn show on the screenshot. 
 
 
Still, I accept some people who need their ego stroked may come into these threads posting, "X is clearly better than Y, check the bass!" and leave.  I cannot accept the idea of someone taking the time to falsify ABX tests, solely to bolster their false claims.  No one could possibly be that pitiful - and if they are, I say we let them continue with the charade, because their self esteem must be INCREDIBLY low. 

 

[Jaywalk3r]

I cannot accept the idea of someone taking the time to falsify ABX tests, solely to bolster their false claims.

Why not? It takes less time to fake results than to actually perform the tests. I doubt very seriously that all of the logs posted are real.

 

[Head Injury]

Here's the funny thing: Most people wouldn't have even thought to cheat until Jaywalk3r brought it up.
 
Jaywalk3r, since you seem to like to crap on blind tests, can you design for us a foolproof one so we can actually gets something done that you'll approve of?

 

[kn19h7]

hmm my experience as a "netizen" tells me no need to doubt ppl on such harmless topic...
 
btw I think practice makes perfect, sites like mp3ornot or klippel.de are good place to start

 

[Jaywalk3r]

Here's the funny thing: Most people wouldn't have even thought to cheat until Jaywalk3r brought it up.

Jaywalk3r, since you seem to like to crap on blind tests, can you design for us a foolproof one so we can actually gets something done that you'll approve of?

I in no way "crapped on blind tests." I'm a fan of properly performed blind tests. I just think it's absurd to only believe someone's test results if they post an easily faked log. It's like a corporation's human resources requiring an ID to verify identity for employment processing, but allowing handwritten "Hi, my name is …" name tags to meet that ID requirement. It makes the requirement meaningless.

If the standard of "proof" is so low, there's no reason we shouldn't simply take someone's word that they ABXed two samples and were able to tell the apart 18/20 times, p-value 0.02%. The attitude that "your comments aren't worth a hill of beans in this thread unless you post logs" is pointless. Logs or not, we're taking posters at their word. There's nothing wrong with that in an informal setting such as this, but it certainly isn't a foolproof system by any stretch of the imagination. These blind test threads, while fun, are not rigorous scientific experiments. The results must be taken with a grain of salt.

If one wants to trust the results of blind tests for a group of people, those being tested should not have access to the results until after they have been formally recorded. The testing process and experimental conditions need to be much tightly controlled. There is a reason authors of studies published in peer refereed scientific journals rarely, if ever, reference Internet forums such as these as sources of credible data. If you want foolproof, the ability to withstand peer review before and after publication is critical.

On the other hand, if you just want entertainment, do some Foobar ABX testing and share with us the results. If you want, you can even post the log. Or not. It doesn't really matter.

 

[Brooko]

Well speaking for myself - I prefer people posted the logs.  I'd like to believe MOST people will have done it legitimately.
 
Of course feel free to keep scoffing at the legitimacy of the results 'if you must'.  I for one don't really care.

 

[Jaywalk3r]

… MOST people will have done it legitimately.

I agree, and never said or implied otherwise.

 

[deadlylover]

@Jaywalk3r
 
Are you suggesting that this track (or 320mp3) is impossible to ABX? So any positive result must be due to a dirty lying scumbag?
 
Because that would be treason against the very concept of lossy encoding itself.
 
Anyway, I can see it's just a misunderstanding, but wouldn't you agree that an opinion posted with an ABX log carries more weight than a one liner?
 
A person who even considers faking the log must have at least done the ABX properly a few times, and if that person is reasonable, they'd also jump on the 'failing bandwagon'. Because, there's nothing to lose, and lets face it, you're supposed to fail.

 
Quote:

see let's all be good head fi samaritans here we shold have a mass blind test thread..but with many different music genres like 3 songs in each genre. and then average the results and see which genre is easier to tell apart from lossless and lossy. im voting on..........modern rock/metal

 
I tried to find a correlation between genre and the ability to be able to successfully ABX lossy, and I've failed thus far.
 
I'd also think that modern rock/metal should be easier, but it doesn't seem so from my limited testing, because with a lot going on, it makes it even more difficult to pick out the artifacts. On the other hand, some simpler tracks seem downright impossible because the encoder has more bits to throw at whatever's there.
 
The only thing I've found is that tracks with very sharp transients or lots of electronic sound effects seem to be the easiest. Killer tracks, as they say.
 
And in my experience, loud listening is absolutely critical (I usually ABX at 90db with my super accurate pain-o-meter), as is proper selection of a slice from the track to test (few seconds max + small soft lead in), because sudden transients can appear to sound very different due to yo' ears trying to compensate/save themselves.
 
kn19h7 said it best, practice makes perfect, because we are automatically accustomed to listening for differences in 'tone' in our day to day lives, and not hunting for barely audible artifacts/differences in attack.
 
All for naught it would seem, because even with the tracks I successfully ABX, the differences practcally vanish under normal listening.

 

[Jaywalk3r]

@Jaywalk3r

Are you suggesting that this track (or 320mp3) is impossible to ABX? So any positive result must be due to a dirty lying scumbag?

I'm not suggesting that at all.

wouldn't you agree that an opinion posted with an ABX log carries more weight than a one liner?

I would not. Why would I believe that someone who is willing to lie is unwilling to edit a snippet of text to support their lie? The log lends no extra credibility.

A person who even considers faking the log must have at least done the ABX properly a few times,

I don't follow your logic. It doesn't require experience with ABX testing to fake the log, only an ability to read and understand the log. I've never done an ABX test "properly." In particular, I've never completed a predetermined number of trials. Once I got bored and quit after a five successes in five attempts, and once I got bored after zero successes in three attempts. That's the extent of my experience with ABX testing.

if that person is reasonable, they'd also jump on the 'failing bandwagon'. Because, there's nothing to lose, and lets face it, you're supposed to fail.

The thing is, this site is visited by many audiophiles and wannabe audiophiles. Many such people have previously claimed how they can hear the difference between lossy and lossless codecs. They might be too embarrassed to post their actual results after they failed the ABX test, but to proud to not post any results. Plus, many people have a competitive nature. There's really only two ways for audiophiles to compete, by having better equipment than the next audiophile or by having demonstrably better hearing. ABX comparisons in this thread and others like it are viewed by some as competitions.

 

[Totally Dubbed]

I shall try this now!

PS. Found this explaining the ABX thingy:
http://www.youtube.com/watch?v=jt7GyFW4hOI
 
 

Spoiler: My%20results%20%3AP

 
LOL - got them all wrong
But also couldn't tell a difference - was really guessing
So i voted for - there is no difference - am i wrong/right?
 
10:48:34 : Test started.
10:50:10 : 00/01  100.0%
10:51:00 : 00/02  100.0%
10:51:55 : 00/03  100.0%
10:52:17 : Test finished.
 
 ---------- 
Total: 0/3 (100.0%) 
 

 

 

[firev1]

 

Spoiler: Results

 
foo_abx 1.3.4 report
foobar2000 v1.1.11
2012/02/17 22:13:30
 
File A: C:\Users\firev1\Downloads\A.wav
File B: C:\Users\firev1\Downloads\B.wav
 
22:13:30 : Test started.
22:14:17 : 01/01  50.0%
22:17:06 : 01/02  75.0%
22:18:15 : 02/03  50.0%
22:19:00 : 02/04  68.8%
22:21:25 : 03/05  50.0%
22:22:25 : 03/06  65.6%
22:22:50 : 04/07  50.0%
22:23:19 : 05/08  36.3%
22:24:13 : 05/09  50.0%
22:26:29 : 06/10  37.7%
22:28:32 : 07/11  27.4%
22:29:22 : 07/12  38.7%
22:29:39 : 08/13  29.1%
22:29:57 : 09/14  21.2%
22:31:52 : 10/15  15.1%
22:33:15 : 10/16  22.7%
22:34:20 : 11/17  16.6%
22:35:36 : 11/18  24.0%
22:36:09 : 12/19  18.0%
22:39:56 : 12/20  25.2%
22:40:11 : Test finished.
 
 ---------- 
Total: 12/20 (25.2%)
 

All I can say is total torture, I count that as a fail really. The clicking instrument seems to hit with a slightly different tonality in A vs B. Otherwise differences are so so subtle that I really can't tell which one is which, I had to turn the treble on my monitors ALL the way up to hear that tiny tiny difference. If I have to pick, maybe A? Voted no diff anyway.

 

[zhenya]

 
 
 


Quote:

I in no way "crapped on blind tests." I'm a fan of properly performed blind tests. I just think it's absurd to only believe someone's test results if they post an easily faked log. It's like a corporation's human resources requiring an ID to verify identity for employment processing, but allowing handwritten "Hi, my name is …" name tags to meet that ID requirement. It makes the requirement meaningless.
If the standard of "proof" is so low, there's no reason we shouldn't simply take someone's word that they ABXed two samples and were able to tell the apart 18/20 times, p-value 0.02%. The attitude that "your comments aren't worth a hill of beans in this thread unless you post logs" is pointless. Logs or not, we're taking posters at their word. There's nothing wrong with that in an informal setting such as this, but it certainly isn't a foolproof system by any stretch of the imagination. These blind test threads, while fun, are not rigorous scientific experiments. The results must be taken with a grain of salt.
If one wants to trust the results of blind tests for a group of people, those being tested should not have access to the results until after they have been formally recorded. The testing process and experimental conditions need to be much tightly controlled. There is a reason authors of studies published in peer refereed scientific journals rarely, if ever, reference Internet forums such as these as sources of credible data. If you want foolproof, the ability to withstand peer review before and after publication is critical.
On the other hand, if you just want entertainment, do some Foobar ABX testing and share with us the results. If you want, you can even post the log. Or not. It doesn't really matter.

No it's not.  Most people in the Sound Science forum are here because it is a respite from the rest of head-fi where 'differences' are waxed on about lyrically for paragraph after paragraph.  We choose to believe that most people here will choose to do the test honestly if they are going to contribute to the conversation and actually care about educating themselves.  The standard that we have chosen to set here is that if you are going to comment, post your logs.  Here's the thing about your attitude - it doesn't matter if some people fake their results.  If we choose to continue these kind of tests here, eventually people will know who they can trust and who they can't, and will make up their own mind whether to believe them based on how they compose themselves here over time.  At least here we have some hurdle in place to take part in the conversation, as opposed to the main equipment forums where suggestions are given every day by people who haven't heard the gear, or who have made only sighted listening tests, often in a noisy meet environment, and where the most respected reviewers are the ones who write the most extensive positive pieces about advertisers gear, which, surprise surprise, gets them asked to do more reviews. 

 
 

 

[Jaywalk3r]

Most people in the Sound Science forum are here because it is a respite from the rest of head-fi where 'differences' are waxed on about lyrically for paragraph after paragraph. We choose to believe that most people here will choose to do the test honestly if they are going to contribute to the conversation and actually care about educating themselves.

There's a difference between talking about science and actually doing scientific research. This forum is a great place to do the former, but limitations inherent to Internet forums make the latter extremely difficult, if not impossible. Ignoring those limitations is in direct opposition to the scientific method.

Here's the thing about your attitude - it doesn't matter if some people fake their results.

That's not my attitude. My attitude is that it's pointless to use logs as a prerequisite to posting a personal conclusion, since the logs are very fast and easy to fake. They won't weed out false claims.

Of course it matters that some people fake their results. That's why these threads are entertainment and not science. That's a limitation of the format (Internet forums), and there's nothing that can easily be done.

 

[Totally Dubbed]

why is it in every second thread i post in, there are people insulting each other...why?