My own personal stance: Technically, it's changing the sound and therefore lower fidelity. Therefore, I do not like the idea on a sheerly emotional / neurotic level.
Cross-feed definitely changes the sound. It would be pointless if it didn't do anything, so of course it
changes the sound. As for the lower fidelity, are you really thinking that, or is it an assumption that all changes to sound are for the worse? Is noise reduction always bad? Is filtering a bass bump away bad? Is removing spatial distortion bad? Did the producers of a recording intent spatial distortion to be part of the listening experience? If so, they fail miserable whenever the recording is listened to with loudspeakers, due to acoustic cross-feed, which by the way changes the sound significantly more than an average headphone cross-feeder does. Sometimes changes to sound are for the better meaning higher fidelity, and cross-feed is a IMO a good example of that.
Also, most mixing engineers (at least for pop / non-audiophile recordings) Will try to produce a mix that sounds reasonable on headphones as well as speakers, so it's not as if headphones are a forgotten realm. Most mixing engineers are resigned to the fact that their recordings will most frequently be heard on iPod earbuds.
You are correct, modern pop is often produced to contain only mild levels of spatial distortion (depends on the producers), but is that all you want to listen to? Weak cross-feed might improve even these recordings taming occasional bursts of spatial distortion. Spatial distortion free recordings do exist and I do listen to them cross-feed off. Knowing how much cross-feed is needed is important for best results (highest fidelity). Sometimes you don't need it at all, but most of the time spatial distortion does exist, even when it's modern pop intended for headphones.
I'm guessing here, but crossfeed should be minimally or not necessary at all for recordings where the soundstage is recorded using a small stereo or binaural setup. Hopefully, a stereo image recorded this way should still sound somewhat natural in headphones, as the recording will still contain the intact stereo image / reverb of the space where the recording was made.
Correct. There are microphone setups that cause very little spatial distortion (such as OSS, ORTF and XY). Binaural recordings should be listened to cross-feed off, but they are pretty rare. Don't cross-feed when there is no spatial distortion to remove! Cross-feed as little as possible to get rid of spatial distortion. It's like adjusting colors on a tv set. You don't want colors pale or over-saturated. You want natural colors. Proper cross-feed means the sound contains just the right amount of spatial information (channel difference).
However, simple stereo setups (such as AB and Blumlein) can produce significant spatial distortion and strong cross-feed is needed to fix things.
When you record with more than 2-3 mics and you're placing instruments artificially and building the soundstage from scratch (this is really a lot more common), the listener may or may not want crossfeed. Hard panning is generally only used as a special effect now, but if you're listening to old Beatles records, you will not hear the recording as intended without crossfeed, because at the time listening equipment was a lot more limited. On the other hand, a fully synthesized electronic music track will not have a "natural" sound no matter what you listen with, as there never was one in the first place.
I pretty much agree with this. However, electronic music can have a very natural sounding spatial image thanks to advanced plugins simulating acoustics for the sound. Spatial distortion is spatial distortion no matter the nature of the music, so fully synthesized music needs cross-feed just as much as totally acoustic music recorded in a real room. If there is spatial distortion, you need to fix it with cross-feed be it jazz, edm, classical or rock.
At the end of the day, if it sounds good, it is good, right? It is usually impossible to know if there is a "more correct" way to play any given recording, but it's not hard to decide whether you like a given effect or not.
My take is that spatial distortion free sound is "correct", because it makes most sense considering how human hearing works and to me it sounds best (natural, realistic, fatigue-free, precise and detailed).