[SonyFan121]

What bigshot done with his test, is the equivalent of playing music through speakers and recording it with a phone, then uploading it on to a computer for playback.
I’m sorry to be brutally honest, but i’m Nobodies fool

 

[bfreedma]

Yes it occurred to me that he might think I would cheat, which I certainly would not do.
The samples where contained within a FLAC audio file which means they where converted from their original format. A very ameture thing to do. Real pro’s would know not to do that, if looking for genuine results.

Bigshot created this test some time ago, so you shouldn’t personalize the methodology. There have been issues in the past with cheating because the testing isn’t supervised - not saying you would, just explaining the rationale.

Can you provide a specific technical explanation of why you believe the conversion to FLAC is problematic in a way that would impact this test? What would you suggest as an alternative that would address the possibility of cheating without using this methodology?

 

[bfreedma]

What bigshot done with his test, is the equivalent of playing music through speakers and recording it with a phone, then uploading it on to a computer for playback.
I’m sorry to be brutally honest, but i’m Nobodies fool

No, it’s nothing like that whatsoever - that’s a false analogy.. I’m trying to give you the benefit of the doubt, but to be brutally honest myself, you seem to be looking for excuses for not being able to differentiate the various formats and bit rates.

In your now edited/deleted post, you thought you had done very well prior to getting the results and raised no issues with the test at that time. Now that you do have the results and weren’t as successful as you believed you were, the test is “flawed”. If you felt the methodology was poor, why didn’t you state that as soon as you received the test file?

 

[SonyFan121]

No, it’s nothing like that whatsoever - that’s a false analogy.. I’m trying to give you the benefit of the doubt, but to be brutally honest myself, you seem to be looking for excuses for not being able to differentiate the various formats and bit rates.

In your now edited/deleted post, you thought you had done very well prior to getting the results and raised no issues with the test at that time. Now that you do have the results and weren’t as successful as you believed you were, the test is “flawed”. If you felt the methodology was poor, why didn’t you state that as soon as you received the test file?

I better not say too much about it. I don’t want to derail this thread somehow. The answer to that would be because I trusted him. It wasn’t until after I took the test that I realised.

 

[Kammerat Rebekka]

So the test was great..until you failed it?
Pretty much sums it up then.

The level of bruised ego and windmill fighting in here is ridiculous.

If you feel Bigshot sidewinded you then try doing it for yourself: rip a cd of your chosing in flac. Upload it in Foobar2000. Then bump it down to 320. Download the free abx program from Foobar as well and once the files have been volumematched you should be good to go.
In this test you will have the opportunity to opt for music that you know like the back of your hand.

 

[old tech]

I better not say too much about it. I don’t want to derail this thread somehow. The answer to that would be because I trusted him. It wasn’t until after I took the test that I realised.

Just wow! Bigshot goes out of his way to assist you with the tests and you repay him by accusing him of dishonesty.

From where I sit, it seems Bigshot has nothing to gain from providing you with the material but you may have a fragile ego to protect when you can't back up your golden eared claims.

Given the serious allegation you are making here, it is incumbent on you to specifically detail why converting lossy files to one Flac file invalidates the test. Given your claims of audio technical knowledge and expertise that should not be hard to do.

 

[StandsOnFeet]

If you feel Bigshot sidewinded you then try doing it for yourself: rip a cd of your chosing in flac. Upload it in Foobar2000. Then bump it down to 320. Download the free abx program from Foobar as well and once the files have been volumematched you should be good to go.
In this test you will have the opportunity to opt for music that you know like the back of your hand.

This is the perfect way to prove to us that you hear what you hear. The ABX plugin lets you listen to your own music as much as you need to make a decision, keeps track of your results, and produces a cryptographically-signed report that you can post here to show real proof of your golden ears. It's easy to set up, and there aren't any "extraneous" conversions to FLAC to confuse the issue. Don't you want to know the truth?

 

[KeithEmo]

There seems to be a common misunderstanding. When you do actual scientific experiments, the ONLY reliable way to test for a certain variable is to control ALL the other variables. You cannot simply assume that certain variables don't matter. By default, we must assume that anything that measurably alters the signal MIGHT produce an audible difference, or obscure one that otherwise exists. Therefore, since we know that the conversion will produce a measurable difference, and we haven't actually tested and proven that converting an AAC file to FLAC will not produce an audible difference, or obscure one that exists, we cannot base the validity of this test on the assumption that it won't do one of those things. In simplest terms, ALL conversions are by definition "problematic", until we determine, on an idividual basis, that a specific one is not.

The best solution is to eliminate any and all variables which cannot be controlled. In this case, we might do that by using some sort of hardware A/B test fixture that can play different file types, without converting them, but without letting the test subjects see which is which. By doing so, we limit the conversions that each file will undergo to the ones we are testing - the conversion to and from each specific lossy encoding option.

The next best solution is to control for our new variable. In this case, we would do that by performing tests to confirm that the conversion and software we propose to use will not produce audible differences of its own, or obscure differences that may otherwise be present. We would do that by performing some benchmark tests that ask people to identify quality differences between known files, and confirm that, if we conduct the same tests, but convert the fiels first, the results remain unchanged.

The least good, but simplest, and most practical, solution is to simply accept that you have failed to control for the variable, and report it in your results as a possible cause for error. There's no great harm in this - and you'll see many drugs which are NOT approved for use with certain age groups, or with pregnant women, simply because they haven't been tested with those groups. HOWEVER, when you do this, you SHOULD be sure to describe this as a possible cause for error in your test results, and it will prevent you from legitimately claiming that your results are universally applicable.

You should also do your best to minimize the risk, and to document it accurately.
In this case.....

- You should carefully document which sample rate or format converter you used and which settings you chose. This allows others to duplicate your experiment under the exact same conditions if they wish to confirm your results. And, if we later find out that certain converters do or don't produce audible differences, or that problems are discovered with specific ones, we will be able to look back and see if that new information affects your reported results.

- In this case, I would suggest delivering the output at a relatively high sample rate, for example 96k. The reason for doing so is somewhat arbitrary, but supported by various facts that are generally true. Most conversions involve some degree of filtering applied to the output files... most often some sort of band limiting applied near the Nyquist frequency. We also know that the sorts of errors produced by most common filter designs often involve frequency response or phase variations near the Nyquist frequency of the sample rate involved. Therefore, without having any specific information, we can infer that we will minimize the likelihood of the conversion producing or obscuring audible variations by choosing a sample rate that puts the Nyquist frequency as far above the audible range as possible. 96k puts the Nyquist frequency well above 20 kHz, while still being supported by most current equipment, so seems like a good compromise there. (We are simply accepting that something is outside our control, and allowing for an extremely wide safety margin - because doing so isn't expecially difficult to do, and will minimize the chances of erroneous results.)

In my personal opinion, GIVEN THAT YOU WANT A TEST WHICH CAN BE OFFERED ONLINE, this probably is the most practical solution, and is the one I would choose. I would also spell out the exact details of each converter I used, including the settings I used, and the software version and operating system. (Converters, and especially lossy encoders and decoders, do change over time - so a different revision of a certain program may well produce different results.)

THERE IS AMPLE PRECEDENT FOR THIS SORT OF DETAIL when it comes to lossy compression - specifically because of the complexity of the process. Because the process is so complex, it has the potential to introduce complex changes. For example, the audio "signal" itself may be reproduced perfectly, but the noise floor may be audibly altered. Many encoders deliberately discard what they consider to be noise; this can result in AUDIBLE pumping or shifts in the noise floor on recordings with a relatively audible noise floor.

(It is well known in video processing that lossy encoders, like the ones used to convert analog video to the format used on DVDs, often produce excellent results with high quality sources, but sometimes produce downright bizarre artifacts with low quality sources.... If you watch DVDs that were created from old analog sources carefully, you will often see an artifact analogous to "breathing", where the noise floor, or film grain, or even movement in dark clouds or shadows, mysteriously appears and disappears, as it shifts above and below the threshold where the encoder decides to discard it as "noise". The result can be quite visible - and quite annoying.)

Bigshot created this test some time ago, so you shouldn’t personalize the methodology. There have been issues in the past with cheating because the testing isn’t supervised - not saying you would, just explaining the rationale.

Can you provide a specific technical explanation of why you believe the conversion to FLAC is problematic in a way that would impact this test? What would you suggest as an alternative that would address the possibility of cheating without using this methodology?

 

[bfreedma]

There seems to be a common misunderstanding. When you do actual scientific experiments, the ONLY reliable way to test for a certain variable is to control ALL the other variables. You cannot simply assume that certain variables don't matter. By default, we must assume that anything that measurably alters the signal MIGHT produce an audible difference, or obscure one that otherwise exists. Therefore, since we know that the conversion will produce a measurable difference, and we haven't actually tested and proven that converting an AAC file to FLAC will not produce an audible difference, or obscure one that exists, we cannot base the validity of this test on the assumption that it won't do one of those things. In simplest terms, ALL conversions are by definition "problematic", until we determine, on an idividual basis, that a specific one is not.

The best solution is to eliminate any and all variables which cannot be controlled. In this case, we might do that by using some sort of hardware A/B test fixture that can play different file types, without converting them, but without letting the test subjects see which is which. By doing so, we limit the conversions that each file will undergo to the ones we are testing - the conversion to and from each specific lossy encoding option.

The next best solution is to control for our new variable. In this case, we would do that by performing tests to confirm that the conversion and software we propose to use will not produce audible differences of its own, or obscure differences that may otherwise be present. We would do that by performing some benchmark tests that ask people to identify quality differences between known files, and confirm that, if we conduct the same tests, but convert the fiels first, the results remain unchanged.

The least good, but simplest, and most practical, solution is to simply accept that you have failed to control for the variable, and report it in your results as a possible cause for error. There's no great harm in this - and you'll see many drugs which are NOT approved for use with certain age groups, or with pregnant women, simply because they haven't been tested with those groups. HOWEVER, when you do this, you SHOULD be sure to describe this as a possible cause for error in your test results, and it will prevent you from legitimately claiming that your results are universally applicable.

You should also do your best to minimize the risk, and to document it accurately.
In this case.....

- You should carefully document which sample rate or format converter you used and which settings you chose. This allows others to duplicate your experiment under the exact same conditions if they wish to confirm your results. And, if we later find out that certain converters do or don't produce audible differences, or that problems are discovered with specific ones, we will be able to look back and see if that new information affects your reported results.

- In this case, I would suggest delivering the output at a relatively high sample rate, for example 96k. The reason for doing so is somewhat arbitrary, but supported by various facts that are generally true. Most conversions involve some degree of filtering applied to the output files... most often some sort of band limiting applied near the Nyquist frequency. We also know that the sorts of errors produced by most common filter designs often involve frequency response or phase variations near the Nyquist frequency of the sample rate involved. Therefore, without having any specific information, we can infer that we will minimize the likelihood of the conversion producing or obscuring audible variations by choosing a sample rate that puts the Nyquist frequency as far above the audible range as possible. 96k puts the Nyquist frequency well above 20 kHz, while still being supported by most current equipment, so seems like a good compromise there. (We are simply accepting that something is outside our control, and allowing for an extremely wide safety margin - because doing so isn't expecially difficult to do, and will minimize the chances of erroneous results.)

In my personal opinion, GIVEN THAT YOU WANT A TEST WHICH CAN BE OFFERED ONLINE, this probably is the most practical solution, and is the one I would choose. I would also spell out the exact details of each converter I used, including the settings I used, and the software version and operating system. (Converters, and especially lossy encoders and decoders, do change over time - so a different revision of a certain program may well produce different results.)

THERE IS AMPLE PRECEDENT FOR THIS SORT OF DETAIL when it comes to lossy compression - specifically because of the complexity of the process. Because the process is so complex, it has the potential to introduce complex changes. For example, the audio "signal" itself may be reproduced perfectly, but the noise floor may be audibly altered. Many encoders deliberately discard what they consider to be noise; this can result in AUDIBLE pumping or shifts in the noise floor on recordings with a relatively audible noise floor.

(It is well known in video processing that lossy encoders, like the ones used to convert analog video to the format used on DVDs, often produce excellent results with high quality sources, but sometimes produce downright bizarre artifacts with low quality sources.... If you watch DVDs that were created from old analog sources carefully, you will often see an artifact analogous to "breathing", where the noise floor, or film grain, or even movement in dark clouds or shadows, mysteriously appears and disappears, as it shifts above and below the threshold where the encoder decides to discard it as "noise". The result can be quite visible - and quite annoying.)

Ok Keith - let me know when you've got that test ready to go and I'll be happy to participate.

 

[KeithEmo]

Everybody should take a step back on this one.....

There is an absolutely valid reason to offer the sample files in the same format - and it is "to prevent cheating".
However, in this context, we're not only talking about deliberate cheating by examining the files in an editor.
We also need to rule out unintentional cues which may consciously or unconsciously tip off the test subject about which file is which.
These range from seeing the file extension or sample rate on your display to unconsciously noticing that one file loads slightly more quickly than another.
We also cannot rule out the possibility that a particular player may offer specific cues - such as making a specific odd little noise, or hesitating a little longer, when starting to play certain file formats.
One reason double blind tests are often used is to rule out the possibility that even a well-intentioned human test proctor may unconsciously "leak clues" to the test subject.
(However, SonyFan121's assertion that the additional conversion introduces a possible point of error is also valid.)

I would also point out that, ignoring the specifics, this certainly demonstrates that misunderstanding about the test protocol itself can easily result in incorrect or inaccurate results.
(We all suffer from a bias that instructions which seem "clear and concise" to us, because we wrote them, may not be as obvious to others, which may lead to errors.)

Yes it occurred to me that he might think I would cheat, which I certainly would not do.
The samples where contained within a FLAC audio file which means they where converted from their original format. A very ameture thing to do. Real pro’s would know not to do that, if looking for genuine results.

 

[Phronesis]

This lossless vs lossy example is turning out to be a good case study illustrating the need to carefully set up, conduct, interpret, and document tests. Looking at it from the outside at this point, I have no way of judging what usefulness bigshot's test has.

 

[KeithEmo]

That's not at all likely to happen....

I personally have better things to do with my time than devise and run expensive and complicated tests. And Emotiva, who I work for, makes audio equipment that works quite well with whatever sorts of files you care to play - so we have no particular motivation to sponsor expensive tests to help you choose which format to use.

As I mentioned in my longer post, the folks who I would expect to be motivated to do that sort of testing would fall into three categories......

1) Companies who sell either software that performs lossy compression or products that rely on it. They would have a vested interest in demonstrating that their lossy compression really is audibly transparent - and perhaps demonstrating that theirs is better than a competitor's similar product or offering. You might suggest this to Apple.... since the existence of iTunes is partially predicated on the idea that "AAC is plenty good for their paying customers, you would think THEY would be eager to prove the truth of that claim.

2) Magazines and other online publications who make a living informing their readers. Obviously it would provide them an opportunity to inform their customers and provide interesting articles on the subject.

3) Local audio clubs or groups. As I mentioned, this would seem like an excellent topic for a public event that might encourage amateur participation and interest.

Ok Keith - let me know when you've got that test ready to go and I'll be happy to participate.

 

[castleofargh]

unless you expect flac to sound different from .wav, there is little cause for concern about lossy being converted to PCM. and the reason is simple, it's going to be done by the player anyway before sending PCM data to the DAC. so the only thing to look for is that the encoder decoder used to extract the lossy file is the same encoder decoder our usual audio player is using. which is always the case for me because I don't bother installing many codecs for a single format and just direct all my software toward the same stuff.

 

[castleofargh]

They are reasonable and rational, but they are still attempting to sell us something more expensive than is necessary to achieve a specific result in an effort to make a profit.

sure I'm admiring Harman's testing efforts and scientific approach, not endorsing all their gears or pricing. although I do happen to have a pair of JBL in front of me, so maybe I'm a little biased .

 

[KeithEmo]

Absolutely agreed.... but only within very specific constraints.

For example, since a FLAC file is lossless, it contains exactly the same data as the WAV file.
Therefore, if we were to play an "original 44k WAV file" and one that had been converted to FLAC, then back to WAV, they will be identical (assuming everything worked right).
And, assuming we were to send both to a DAC, the DAC should be receiving the same PCM audio in both cases (the player should be converting both files to the same PCM output data).
Although we have introduced the possibility that, because decoding FLAC takes more processing power than playing a WAV file, the computer might introduce errors during the conversion.
(We can test for this, and confirm that it isn't happening, by checking that the output of the computer is still bit-perfect.)

However, sample rate conversions always entail some filtering, especially when we're talking about uneven multiples.
So, for example, if you convert a 44k WAV to a 96k WAV, filtering will be applied, so there WILL be tiny measurable differences (and they will depend on the settings you pick in the converter).
And, now, lets say you start with a 320k VBR MP3 - which is simply a different format.
If you decode that file, using a certain MP3 decoder, with the target format as 44k WAV, you will get a certain result.
Then, if you convert that 44k WAV file to a 96k WAV file, it will be altered slightly by the conversion process.
So if, instead, you decode the MP3 file directly to 96k WAV, using the MP3 decoder, the result will be slightly different.
You omitted the conversion from 44k WAV to 96k WAV, so you avoided that difference, but we don't know if that particular MP3 decoder may produce different results based on the output sample rate you chose.
(It's not at all unlikely that part of the decoding process may include filters which are chosen in part based on the Nyquist frequency of the chosen output sample rate.)

Another thing that needs to be considered is that many lossy encoders are what I would term non-deterministic by process.
For example, the MP3 DECODER is standardized, meaning that, if you play the same MP3 file using different decoders, you SHOULD get the same audio output.
However, the performance of the MP3 ENCODER is NOT FULLY SPECIFIED.
The specification essentially requires the encoder to produce a file that will play properly on the standard decoder (assuring that any MP3 file will play on any MP3 player).
However, within the broad limitations of "the standard model", each individual MP3 ENCODER has "discretion" about what information to discard.
So, if you encode the same original file on two different MP3 encoders, using the same exact settings, the output will be different, AND MAY SOUND AUDIBLY DIFFERENT.
(Individual results are deterministic. If you encode the same file, using the same encoder, and the same settings, you will get the same result.)
The standard ensures that all MP3 files will play on your standard decoder - but it in no way ensures or even suggests that they will be the same.
(In fact, the standard was designed that way to allow, or even encourage, "improvements in the encoding technology".)
This specifically suggests that you should expect differences between encoders, and even between different versions of the same encoder, or the same encoder on different types of computers.
(It is worth noting, however, than most commercial programs use one of two or three readily available encoder program modules internally, so many do in fact produce identical results.)
(I don't know how this applied to AAC... although I believe the situation is similar.)

It's also not a good idea to simply assume that ANY software works the way it should.
For example, while performing a reasonably accurate sample-rate conversion is mathematically not especially difficult, some software still manages to do it poorly.
There is a site that reviews various sample rate converters: http://src.infinitewave.ca/
By the metric they chose, about half of the commercial SRC products available seem to do an accurate job of converting between sample rates with few noticeable artifacts, but the other half do a far less accurate job.
(Therefore, unless you actually test the particular one you plan to use, it is not safe to assume that a particular converter will perform well.)
Likewise, if you're planning to use FLAC files, it's not a bad idea to confirm that the converter you're using is converting them properly as well.
(Programmers make mistakes; and certain vendors may deliberately introduce colorations in a player or converter product that they believe constitute "an improvement".)

unless you expect flac to sound different from .wav, there is little cause for concern about lossy being converted to PCM. and the reason is simple, it's going to be done by the player anyway before sending PCM data to the DAC. so the only thing to look for is that the encoder decoder used to extract the lossy file is the same encoder decoder our usual audio player is using. which is always the case for me because I don't bother installing many codecs for a single format and just direct all my software toward the same stuff.