http://rationalwiki.org/wiki/Dunning-Kruger_effect you can see this in reviews or anywhere else. a few times I went head to head with a review that was IMO a total nonsense(not on headfi I would get banned for saying the stuff I said
). the most obvious thing every time is that the guy has no idea he's doing something wrong. he just got his new toy and like with each toy, he's overjoyed for 2 or 3 weeks and makes a review of how happy he is. nothing to do with the device tested, most of the time he could have bought a new watch and tell us how much it improved the soundstage for all that matters.
we can usually catch those guys with how overly positive they are if we also tested the device, but if we didn't, IMO a good cue is to look at how fast his brilliant gear will be sold and replaced by another so called brilliant gear.
and the other way around, I often discuss with people that have a pretty good hearing and understanding of where a device stands compared to the competition, but they will never do a review because they believe they're not qualified to do one(which really is our loss IMO).
I'd like to +100000 the EQ part. how can a guy explain the sound of a device if he can't identify a 5khz spike or where the signal rolls off? even the EQ haters, and we know it's still the majority of people here, should practice with a nice EQ on the device being tested as a way to identify the strength and flaws of the signature. just going "on the song XXXXX the second guitar is a little less vivid than when I use that other device". that will help 3 people who have at least the other device and will try the song. but if we talk about frequency response, then the guy can go move that frequency on his own system and understand what we're talking about pretty much all the time. an EQ is a great tool for that job IMO. when you talk about how the trebles are too strong with someone and you realize his idea of trebles was 3khz, Houston we have a problem.
about the stars I have no idea what I'm doing! and most of the time I guess I'm just rating if I found it was worth it in a very very insecure way ^_^. also I wouldn't review something I would rate 1 or 2 stars, so in my case the stars have close to no meaning.