Even when encoding, although you are using the same source file, the encoder itself will often have the transcoded file at a slightly different volume level to the original. And as you're aware, even 0.5 dB difference is enough to tell two files apart if your hearing is reasonably acute. So that's issue number one. The second issue is using the iPod to A/B - it doesn't have the software to do it. The easiest way is as per my guide - Foobar 2000 + abx comparator + built in volume matching + 15 iterations of each track tested. It really is an eye opener.
For anyone wanting to know more - I know you've seen the guide MA - link is here : http://www.head-fi.org/t/655879/setting-up-an-abx-test-simple-guide-to-ripping-tagging-transcoding
One of these days you'll get the chance to try it. I can promise you that if you've set everything up correctly, it will change your mind about what is really audible
hehe...I went to bookmark the link, but it had already been bookmarked.
I actually had someone create a custom command line encoder for me in order to use QuickTime AAC with dBpoweramp.
iPods support ReplayGain, which seems to be all you propose in order to do volume matching.
The Rockbox firmware presents more opportunities - though I don't know if it's customizable enough to conduct this type of test.
Here's a crazier question that is somewhat relevant to this thread: since AIFF files reverse the byte order, do you think it's possible for them to sound different than WAV files under certain circumstances, despite both of them being uncompressed? It's just that many people (including famous studio professionals with decades of experience) claim to hear differences between lossless formats, and if this is so, it would only be due to how different devices process the data, which is identical after decoding. I had discounted all of this for a while, but recently began comparing again...and AIFF files seem to sound slightly more harsh and less natural than WAV when played back on my iPod Classic. The most likely scenario is that it is simply my imagination playing tricks on me, but others have reported this phenomenon as well, so I feel it's worth looking further into. If only I had the equipment to measure its analog output while playing the files...