Another important but often overlooked effect is that of the grille frame on the sound - in particular as regards to diffraction. In many cases I would expect this to have a bigger effect on the sound of the loudspeaker and may in fact account for much of the apparent difference in clarity.
A soft grille cloth, only absorbing a very small portion high frequency waves and with almost no reflection, should have minimal impact on the sound. By comparison, a rectangular MDF frame around the speaker will cause several secondary reflections, muddying sound arrival times (to what degree I do not know) and certainly affecting frequency response.
On the other hand, some grilles are designed to minimize or even improve diffraction behavior.
My main speakers, a pair of Infinity Renaissance 90s, incorporate this. The manual explicitly states that the grilles are necessary for proper performance. The drivers are raised above the cabinet face so that they are flush with the grilles. The grilles are solid MDF and taper at the sides (matching the cabinets) to minimize diffraction. In fact, there was a particular grille option that didn't even have fabric, for people that wanted to see the drivers. See the photo below:
And here they are without grilles:
Unfortunately, my grilles were broken by the previous owner and so I have never heard them with the grilles on. I'd like to have replacements made out of stained high-grade plywood though, with no cloth.