[ClieOS]

Yep, the link to F is on the first post.
 
The differentiated clips I mentioned earlier are generated via Audio DiffMaker as well. The result is interesting to say the least, but I can foreseen it could create some kind of bias to the subjective opinion. Hence why I'll strongly suggest you guys do a listening first and note down whatever you have heard before doing any software analysis. Also, please keep the software analysis result to yourself for now and see if they lineup with everyone else when the result is published.
 
If I have time, I'll generate an additional 6 clips using the same condition as the original 6 clips, probably with some live orchestra recording since they tends to have better dynamic, just to give a better contrast.

 

[ClieOS]

As promised, here are 6 more music clips, about 1 minute each, for another round of comparison. They are produced using near identical condition as the previous 6 files. The difference is that these are live recording / orchestra work so hopefully they will offer higher dynamic range.
 
http://www.mediafire.com/download/928vsjzc0es1kp2/MusicComparison2.zip

 

[ClieOS]

As far as I know, all the changes made between X5's firmware v2.0 to v2.2 do not involve anything that will affect the SQ. Yet there are occasional discussion that suggests some do hear a difference from one firmware to the next - thus the reason to generate all these music clips for comparison. They are all recorded from X5's line-out into Focusrite Scarlett 2i2, volume matched so the recording is as close to X5's original volume level as possible. Recording were done in Audacity, export to wav files then convert to FLAC.
 
Now it is time to reveal the answer:
 
Music clips - Firmware version
A - v2.2 final
B - v2.02 beta
C - v2.0 final
D - v2.04 beta
E - v2.2 final
F - original FLAC
 
G - v2.2 final
H - v2.02 beta
I - v2.0 final
J - v2.04 beta
K - v2.2 final
L - original FLAC
 
Basically, A and E or G and K are recorded under the latest firmware, to act as an control. F and L are the original copies, simply chopped to the right length as the recorded sample. If you find either to be better sounding than the rest, then you are correct. The big question however is, does any of the firmware version sounds different from the others? For that, we need a help from a software call Audio DiffMaker. I won't be bother with you for all the detail about this software as there is a published paper you can read on its page. In simple term, this software takes two pieces of recording, find out the difference and make it into an audio "subtraction" clip. The process is not totally perfect, but if the resulting clips is very low in volume, it means the two original clips are extremely similar and can be concluded to have no audible difference to the human ears.
 
The result is, all recording via different firmware should make no audible difference, as the differences generated using different clips on average go from -55dB to as low as -72dB (counting into the error generated by Audio DiffMaker itself), which are all below what human can detect between different tracks. In contrast, the differences generated using the original FLAC and the recording are within -20dB range for the first 6 clips and within single digit dB for the last 6 clips, and well within detectable range. The reason F is less differentiable to A~E while L is more differentiable to G~K is because the music used in L has better dynamic range as a music and therefore any difference between original file and the recordings are more apparent.
 
Below are two samples of the subtracted clips
 
http://www.mediafire.com/download/2roikp7456fvc2b/2.2-2.02.flac
^ this is A vs. B. Comparison between A~E are all pretty similar.
 
http://www.mediafire.com/download/vmrffrxtk9qubqu/Ori-2.2.flac
^ this is A vs. F

 

[castleofargh]

yeah!!!! I got working ears!!! mum will be so proud. did I won a check like this one?

 
 
thank you for making this, even if posting into sound science was sadly a sure way not to be seen by most headfiers. I at least learned that the X5 seems to sound very fine even after going through your own recording system. nice to see that fiio keeps improving each time.

 

[musicheaven]

I have inverted and mixed G,H, I and J together (did not have the patience to do them all) and also visually inspected the audio envelope from those 3. The end results is minimal and the mixing exercises provided a flat response file meaning as far as they are concerned they are identical. I also listened to all 4 and could not discern any differences. Are you sure those are from different firmware revisions? 

 

 

[ClieOS]

  ...Are you sure those are from different firmware revisions? 

 

 
That's the question

 
Of course, there is always the possibility that my X5 is broken.

 

[musicheaven]

   
That's the question

 
Of course, there is always the possibility that my X5 is broken.

Great work, certainly makes you think and wonder.

 

[issachar1943]

  There is always debate of whether firmware update affects sound quality for just about every DAP / software / gears. Below is a link to download a zip file can contain five 30 seconds music clips that were played by FiiO X5 on different firmware version, then recorded in identical setting (Focusrite Scarlett 2i2 + Audacity into PCM wave, then converted to FLAC). See if you can tell any difference. If you can, tell us which one is better / worst. It shouldn't matter if you use X5 for playback or not, but do try to use your most revealing gears.
 
http://www.mediafire.com/download/3kd1h8hew8c5lae/MusicComparison.zip
 
Firmware version will be revealed in a week.
 

---

 
Just for extra fun of comparison, please add the music clip below to the above comparison:
 
http://www.mediafire.com/download/kk9aai8s4uwl4kk/F.flac
 

---

 
Update:
 
Here are 6 more music clips, about 1 minute each, for another round of comparison. They are produced using near identical condition as the previous 6 files. The difference is that these are live recording / orchestra work so hopefully they will offer higher dynamic range.
 
http://www.mediafire.com/download/928vsjzc0es1kp2/MusicComparison2.zip
 

---

 
Now it is time to reveal the answer:
 
Music clips - Firmware version
A - v2.2 final
B - v2.02 beta
C - v2 final
D - v2.04 beta
E - v2.2 final
F - original FLAC
 
G - v2.2 final
H - v2.02 beta
I - v2 final
J - v2.04 beta
K - v2.2 final
L - original FLAC
 
Basically, A and E or G and K are recorded under the latest firmware, to act as an control. F and L are the original copies, simply chopped to the right length as the recorded sample. If you find either to be better sounding than the rest, then you are correct. The big question however is, does any of the firmware version sounds different from the others? For that, we need a help from a software call Audio DiffMaker. I won't be bother with you for all the detail about this software as there is a published paper you can read on its page. In simple term, this software takes two pieces of recording, find out the difference and make it into an audio "subtraction" clip. The process is not totally perfect, but if the resulting clips is very low in volume, it means the two original clips are extremely similar and can be concluded to have no audible difference to the human ears.
 
The result is, all recording via different firmware should make no audible difference, as the differences generated using different clips on average go from -55dB to as low as -72dB (counting into the error generated by Audio DiffMaker itself), which are all below what human can detect between different tracks. In contrast, the differences generated using the original FLAC and the recording are within -20dB range for the first 6 clips and within single digit dB for the last 6 clips, and well within detectable range. The reason F is less differentiable to A~E while L is more differentiable to G~K is because the music used in L has better dynamic range as a music and therefore any difference between original file and the recordings are more apparent.
 
Below are two samples of the subtracted clips
 
http://www.mediafire.com/download/2roikp7456fvc2b/2.2-2.02.flac
^ this is A vs. B. Comparison between A~E are all pretty similar.
 
http://www.mediafire.com/download/vmrffrxtk9qubqu/Ori-2.2.flac
^ this is A vs. F

 
This is extremely interesting! I am downloading and will soon found out if my brain is tricking me. 

 

[ClieOS]

   
This is extremely interesting! I am downloading and will soon found out if my brain is tricking me. 

 
Well, you already know the answer so there is no easy way to avoid bias, even if you are not aware of it and don't think you are. But it is still fun to try no less.

 

[darkarn]

Whoa, thanks mate, did the test before your results and found them to sound the same!
 
I wonder if the different listening environments may have impacted their results...

 

[issachar1943]

   
Well, you already know the answer so there is no easy way to avoid bias, even if you are not aware of it and don't think you are. But it is still fun to try no less.

 
Yes, I see your point. Tried my best to avoid bias.
 
Now my question is, are those answers real? 
 
Okay, I must admit at first few times, G and H sounds little different. But after listening to them repeatedly, they sounds mostly identical.
 
Also, is it that theoretically, the original file L should sound different as it's not being processed? It really confuse me now or am I deaf? hahaha 

 

[castleofargh]

 

   
Well, you already know the answer so there is no easy way to avoid bias, even if you are not aware of it and don't think you are. But it is still fun to try no less.

 
Yes, I see your point. Tried my best to avoid bias.
 
Now my question is, are those answers real? 
 
Okay, I must admit at first few times, G and H sounds little different. But after listening to them repeatedly, they sounds mostly identical.
 
Also, is it that theoretically, the original file L should sound different as it's not being processed? It really confuse me now or am I deaf? hahaha 

do an ABX, I can listen to the same track and think I found a difference. to the point where I'm confident I will be able to tell what is what the bass is this the soundstage is that... then I ABX and after 10 or 20 tries I end up with a magnificent 50/50 despite my confidence ^_^(that's why so many people refuse ABX because they're too proud to admit they fooled themselves). if you look for differences chances are you will find some even if there is nothing.

 

[ClieOS]

   
Yes, I see your point. Tried my best to avoid bias.
 
Now my question is, are those answers real? 
 
Okay, I must admit at first few times, G and H sounds little different. But after listening to them repeatedly, they sounds mostly identical.
 
Also, is it that theoretically, the original file L should sound different as it's not being processed? It really confuse me now or am I deaf? hahaha 

 
At initial listening, I always found v2.04 to be different from the rest and confident that I should be able to pick it out in an ABX - then I can't. That's how your mind can trick you into believing what is not there.
 
Any playback-recording process will definitely lose some detail. The better then equipment, the less lost you will suffer. I have to admit how good the Fucosrite is. I don't actually use it for recording but only for measurement. But given its price, it is still a very good bit of kit. Don't feel sorry if you can't tell the difference. If anything, human is known to have very short audio memory. It is perfectly normal for untrained ears to not notice any difference if the tracks are sufficiently similar.

 

[stv014]

Originally Posted by ClieOS /img/forum/go_quote.gif

In contrast, the differences generated using the original FLAC and the recording are within -20dB range for the first 6 clips and within single digit dB for the last 6 clips, and well within detectable range. The reason F is less differentiable to A~E while L is more differentiable to G~K is because the music used in L has better dynamic range as a music and therefore any difference between original file and the recordings are more apparent.

 
This is most likely an Audio DiffMaker problem, when comparing the recorded audio to the original, there is a relatively large ~75 ppm pitch error, which confuses the program, and it does not extract the difference correctly. Also, correction for sample rate drift needs to be enabled in the settings, but even if it is, it is not measured accurately, as DiffMaker only finds about a tenth of the real value. With some manual correction of delay, pitch, and amplitude (resample.exe -isrm 0.99992461175418321850 -k 0.00200171041960997732 -g1 1.13040505535016335435 -g2 1.12862598871029550435 -f 2 -d 59.95047619047619047619 G.wav G_.wav), the difference between G.wav and L.wav goes down to under -30 dBr, but it is still not good, and DiffMaker does not make much improvement over simply subtracting the files without equalization.
 
In this type of test, I tend to add an MLS loop and a few seconds of tones before and after the sample, to allow for better manual extraction when Audio DiffMaker fails to work correctly. Although it could also be fixed with better settings after some experimenting.

 

[ClieOS]

Good to know. I am really not that familiar with Audio DiffMaker to make that much fine tuning.