The blame goes far beyond Diederik Stapel and the three Dutch universities where he worked as a social psychologist. In their exhaustive final report about the fraud affair that rocked social psychology last year, three investigative panels today collectively find fault with the field itself. They paint an image of a "sloppy" research culture in which some scientists don't understand the essentials of statistics, journal-selected article reviewers encourage researchers to leave unwelcome data out of their papers, and even the most prestigious journals print results that are obviously too good to be true.
The commissions—one for each of the universities where Stapel has worked—concluded that "from the bottom to the top there was a general neglect of fundamental scientific standards and methodological requirements." That climate made it possible for Stapel's fraud to go undetected for many years, the report says.
Entitled Flawed science: The fraudulent research practices of social psychologist Diederik Stapel, the report again casts an unflattering light on social psychology, which has seen several other investigators come under scrutiny since Stapel was fired from his post at Tilburg University in September 2011. Two months ago, psychologist and Nobel laureate Daniel Kahneman urged researchers doing so-called priming studies—an important area in social psychology where Stapel was also active—to clean up their act, declaring in an open letter that "your field is now the poster child for doubts about the integrity of psychological research."
Over the past 13 months, the panels investigating Stapel combed through all of his 137 papers and interviewed more than 80 people; two of the three panels also talked to Stapel himself. The work was coordinated by psycholinguist Willem Levelt, a former president of the Royal Netherlands Academy of Arts and Sciences who chaired the Tilburg commission.
In a final joint report, the commissions conclude that Stapel committed data fraud in 55 papers and in 10 Ph.D. theses written by students he supervised. In another 10 papers, scientific misconduct could not be determined beyond a reasonable doubt, but there was "evidence of fraud," such as incorrectly reported p-values or numbers of participants, the report says.
Some of the factors that helped Stapel perpetrate the fraud had been identified in an interim report last year. Stapel often designed studies with others, but then said he'd collect the data himself. He had the reputation of a "Golden Boy" who was almost beyond criticism; he cultivated personal friendships with his students—many of whom found him an inspiring and charismatic leader—but did not tolerate critical questions about his data. Colleagues left him alone; the three young researchers who eventually blew the whistle "showed more courage, vigilance and inquisitiveness than incumbent full professors," the report says.
But even in Stapel papers that weren't outright fraudulent, the commissions found plenty of serious flaws that point to a culture of "sloppy" science. They noticed examples of verification bias, in which, for example, an experiment is repeated until it produces the desired outcomes, or unwelcome experimental subjects or results are thrown out; the investigators stumbled upon cases in which the research procedures described in a paper were different from those actually used; and statisticians on the panels found "countless flaws" and discovered that Stapel's co-authors seemed to have a "lack of familiarity with elementary statistics," paired with a "lack of interest" in them.
In one example that the report cites, estimated data "obtained from some form or other of smoothing or curve fitting" was presented as the actual raw data. Stapel's co-author did not see the problem with that, the report says; he or she didn't know how it happened but guessed it was done " 'by using an option in Excel.' "
Although it stops short of saying these cases are representative of the social psychology field as whole, the fact that Stapel's "blatant" fraud was never caught seems to speak volumes to the panels:
It is almost inconceivable that co-authors who analysed the data intensively, or reviewers of the international "leading journals", who are deemed to be experts in their field, could have failed to see that a reported experiment would have been almost infeasible in practice, did not notice the reporting of impossible statistical results, … and did not spot values identical to many decimal places in entire series of means in the published tables. Virtually nothing of all the impossibilities, peculiarities and sloppiness mentioned in this report was observed by all these local, national and international members of the field, and no suspicion of fraud whatsoever arose.
Psychology journals and their reviewers get their share of the blame, because they seemed to invite Stapel and his co-authors to massage the data:
Reviewers have also requested that not all executed analyses be reported, for example by simply leaving unmentioned any conditions for which no effects had been found, although effects were originally expected. Sometimes reviewers insisted on retrospective pilot studies, which were then reported as having been performed in advance. In this way the experiments and choices of items are justified with the benefit of hindsight.
Not infrequently reviews were strongly in favour of telling an interesting, elegant, concise and compelling story, possibly at the expense of the necessary scientific diligence.
Stapel released a written statement (in Dutch) today to the press, which he also delivered in a 2-minute video recorded by Dutch public television. He does not address the report directly, but says:
I feel deep, deep remorse for the pain I have caused others. I feel a great deal of sadness, shame and self-blame. The truth would have been better off without me.
I have created a world in which almost nothing ever went wrong, and everything was an understandable success. The world was perfect: exactly as expected, predicted, dreamed. In a strange, naive way I thought I was doing everybody a favor with this. That I was helping people. …
The past year and a half, I have made an intense effort to understand and deal with my behavior. "How in heaven's name could this have happened?" I have sought professional help, I started writing diaries, and thanks to conversations with friends, relatives, and to chemistry and therapy, I am learning step by step to stare my demons in the eye and tame them.
Stapel also said he will publish a book later this week containing diary notes from the past year. Dutch media reported today that it will be called Ontsporing (Derailment), in an apparent reference to Ontspoorde Wetenschap (Derailed Science), a recent book by Dutch journalist Frank van Kolfschooten in which Stapel features prominently.