Nope, I mean RF radiated noise. Even if the switching frequency is, say, only 500kHz, you're talking a 500kHz square wave at up to 100-200V P-P depending on the power output of the Class D amp. 500kHz square waves at those amplitudes are going to have significant harmonics out into the tens of megahertz--which gives you the potential to take out AM radio reception, interfere with FM radio, cause problems with old-style RF interfaces--and, more importantly, be picked up by other electronics, rectified and folded back as noise or distortion.
This kinda stuff is the kinda stuff the FCC does NOT like. Any device with a clock greater than 9kHz must be FCC Class B (consumer) compliant. Doing it with 3.3V P-P signals is sometimes problematic. Doing it with signals 30X that amplitude, especially when connected to unshielded (speaker) cables, doesn't thrill me.
If you take a look at the radiated noise of a Class D amp or a switching power supply on an RF spectrum analyzer, and then look at the ground noise of the same devices on a highly resolving instrument (today's digital scopes aren't all that great for this--I'm talking something like a Tektronix 7000-series mainframe oscilloscope with a 7A22 differential plugin, which gets down to 10nV/division) you would never want to deal with them, unless it's for an application where you're forced to use them, with space and power limitations, like portable devices.
Linear is our future.