TLDR?
One more observation on blind testing. A frequent “requirement” passed on by the demanders, is “no discussion”, for fear that people will influence each other. I disagree. I work with technical people and groups usually make better decisions than individuals. Not always, but usually. I had an experience recently that bears on this.
I test large industrial stuff, I won’t mention the product or the company. I am not an expert in anything but I am an expert in “getting all the right experts there, and getting them to agree.” We had a product in test, I was not present, it belched flame out of a vent where no flame was expected. The first response in the room was, “DUDE!” and the second response was, “Schiit, shut it down, we broke it!” But inspection revealed no broken parts so we carefully ran it again. It did not sound funny or smell funny and the output was good, so we carefully repeated the test point that had produced a flame. The flame reappeared. We called in the chemical kinetics team, they are both really smart and really frustrating. They thought for a day or so and then patiently explained that what we saw was not flame, it was simply hot gas with glowing dust and soot. Glowing yellow orange because it was really hot. “That sounds like flame” I said, but apparently it just looks like flame. Don’t stick your hand in it, regardless. The chemical kinetics experts prescribed a series of tests to see where and when this non-flame occurred. We hired a guy with a super expensive infrared camera to photograph the entire test. He was so expensive that we only got him for one day. He set up his camera looking at our equipment and set up his computers in a spare room down the hall and we forgot about him. He was so introverted that he didn’t even come out for pizza. (We invited him!)
There were about ten of us in the room, engineers, mechanics, a crane operator, a PhD Chemist, a co-op who still doesn’t shave, a rising young executive. We ran the test through several test points and everyone either saw the non-flame or they didn’t see it. Until one test point. Everyone said, “Nothing” except for two people who said, “I saw it.” We reran that point, everyone but two guys saw nothing, those two still saw something. We argued a bit until I remembered the guy with the fancy camera, who was getting actual numerical readings of infrared energy. (That is where I earned my pay.) We visited him. He was seeing a whopping infrared “bloom” on our test points with the visible “flame.” We asked about the last two test points? They had that same infrared bloom but it was diminished, he said, “maybe half the total energy.” So we finished the test and reported some points with a visible effect, some with none, and two that had “some visibility.” The expensive guy with the expensive camera went away. On the next test we went in expecting mixed results and planning to “vote.” We actually found more tests point with mixed results, and were mildly surprised when more people saw “low level” stuff. The team spontaneously started rating the observations as “Yes we saw it”, “No, nothing” or “partial visibility.” Partial meant a few saw it and a few didn't. I believe that the interactions and discussions made people more observant, not less. No one had any ego involved, we were just watching glowing gas. And the guy who could actually measure stuff helped us make a decision about what we saw. The chemists were able to use all this to solve the problem.
So I think blind testing should have open minded discussion between people who are willing to argue constructively. With the possibility to demand replays. And a willingness to respect the other person even when a result is unexpected.
