My own tests have varied wildly from that.
I used some of the highest quality lossy formats (such as 320 kbps MP3 and Q127 QuickTime tVBR AAC) and listened to lossless and lossy encodings of dozens of albums across many genres.
On about 25% of the music I tested, I did not notice any differences at all.
However, on the remaining 75% or so, the differences were either small, but significant enough to notice...or painfully obvious.
In the more extreme instances, audible distortion manifested itself in the form of bloated bass, sibilant treble, diminished dynamics, and all the other usual stuff most people would only notice in poorly encoded MP3s.
For most listeners, lossy is fine; I make it a point to listen in CD quality unless that isn't available.