I am writing to you with my own (very) small story of error-checking a published finding. If you end up posting any of this, please remove my name!
A few years ago, a well-read business journal published an article by a senior-level employee at my company. One of the findings was incredibly counter-intuitive. I looked up one of the reference studies and found that a key measure was reverse-coded (e.g. a 5 meant “poor” and a 1 meant “excellent”). My immediate conclusion was that this reverse coding was not accounted for in the article. I called the author and suggested that with such a strange finding, they should check the data to make sure it was coded properly. The author went back to the research partner, and they claimed the data was correct.
Still thinking the finding was anomalous, I downloaded the data from the original reference study. I then plotted the key metric and showed that the incorrectly coded data matched what was published, but that the correctly coded data matched the intuition. I sent those charts to the senior person. The author and research partner double-checked and confirmed there was an error in their reporting. So far so good!
After confirming the error, the author called and asked me “What are you trying to accomplish here?”. I responded that I was only trying to protect this senior person (and the company), because if I found the error somebody else would find it later down the line. The author, however, was suspicious of why I took the time to investigate the data. I was puzzled, since it appeared it was the research partner who made the fundamental error and the author’s only fault was in not diving into a counter-intuitive result. In the end, the graph in question was redacted from the online version of article. And, as you by now would certainly expect, the author claimed “none of the conclusions were materially impacted by the change”.
Do you have a name for this phenomenon in your lexicon yet? Might I suggest “eureka bias”? Meaning, when somebody is well-intentioned and discovers something unique, that “eureka moment” assumes a supremely privileged status in the researcher’s mind, and they never want to abandon that position despite evidence to the contrary…
Source : http://andrewgelman.com/2018/05/16/eureka-bias-think-made-discovery-dont-want-give-even-turns-interpreted-data-wrong/Thank You for Visiting My Website