A few points:
Time - I turn on my CD player cold and press play, I've got audio in under 10 seconds. I've never seen a computer that can cold-boot that fast. Vista can return from sleep mode that fast, which is a notable improvement over every other OS I've used, but sleep mode and a cold start aren't the same thing. Plus then I have to start the media player, find the song I want, make it play, etc.
Noise - It's difficult and tedious to make a computer silent, and especially with hard drive noise and vibrations you usually won't get around some background hum. Even my cheap CE-595 is silent while playing.
Remote Control - Yes, you can theoretically get one with your computer, but that's added cost, configuration time, and hoping it works with both your OS and your media player of choice. The one with my CD player comes stock, is compatible with universal remotes, and requires no setup.
No Background Noise - I won't have pop-up ads screaming about Viagra right at the climax of a Mahler symphony. I also won't have problems with different applications being at different volume levels.
It's nice to have my music library digitized - it's a joy to have it on my iPod, and even on the computer it's occasionally convenient. But when I want to listen to music, 19 times out of 20 I press play on the CD player, not on foobar2000.