Yeah people who say there's a big difference seriously need to do ABX. In most case where people can tell the difference, it as an above poster put it, in absolutely perfect conditions with a track you're very familiar with, and even then you'll still guess wrong some of the time.
I used to be one of those "of course there's a difference" people... until I did ABX, and found I had to repeat very specific portions of songs over and over again to tell the difference, and even then I would be wrong about 20% of the time.
The funny thing is when I do the test in a non-blind fashion, the FLAC sounds better in every way. There seems to be better spatial definition, more ambiance, better highs, tighter bass. But as soon as you blind yourself to which is which, it's amazing how those differences almost all disappear.
It's also quite sad when you hear people say "FLAC is almost as good as CD." when they are exactly the same thing. These are the same people who believe putting CDs in the freezer makes them sound better.
The human mind is easy to trick.
You have to do a blind ABX test, or otherwise you're deluding yourself.
Of course, I'm talking about well-encoded lossy formats. Apple's 256 AAC is certainly well-encoded. LAME 320 and v0 (or even v3) is also excellent.
If you're downloading 320 files from random places on the web (like mp3skull), be aware lots of those files aren't encoded from proper lossless sources. It's easy to rip a song from youtube and encode it 320. Of course in that case it's not gonna sound so good.