Bem's "Feeling the Future" experiments

The full file

In March 2011 the Journal of Personality and Social Psychology, published by the American Psychological Association, printed a paper by Daryl Bem, a social psychologist and emeritus professor at Cornell University. Titled "Feeling the future", it reported nine experiments, run over about a decade with more than 1,000 student participants, which appeared to show that people's responses could be influenced by events that had not yet happened. Advance copies had circulated among psychologists in late 2010, and the New York Times reported the coming paper on 5 January 2011, quoting the editor, Charles Judd, that four reviewers had passed it.

What was reported

Bem's method, in the abstract's words, was "time-reversing" well-established psychological effects. In the best known, Experiment 9, students saw a list of 48 words, typed as many as they could recall, and only then practised a random half of the list. In ordinary memory research practice helps recall; Bem reported that the words practised afterwards had been recalled better. He called this "retroactive facilitation of recall". Others reversed priming, habituation and the pull of erotic pictures. By the paper's abstract, the mean effect size across all nine was d = 0.22, and all but one experiment gave statistically significant results. Participants high in stimulus seeking did better, with a mean effect size of 0.43. The paper invited replication, and Bem offered the software needed to run it.

The record and where it is held

The record that puts this case on the Investigated shelf is the published literature itself, made at the time and largely by people other than the author. Bem's paper (volume 100, issue 3, pages 407–425) appeared in the same issue as a critique by Eric-Jan Wagenmakers, Ruud Wetzels, Denny Borsboom and Han van der Maas of the University of Amsterdam (pages 426–432). Both are held in the APA's journal archive. The first pre-registered replications, by Stuart Ritchie (Edinburgh), Richard Wiseman (Hertfordshire) and Chris French (Goldsmiths, London), were published in the open-access journal PLoS ONE on 14 March 2012. A larger set by Jeff Galak, Robyn LeBoeuf, Leif Nelson and Joseph Simmons appeared in the Journal of Personality and Social Psychology itself in 2012, online in August and in the December issue. Bem and three colleagues' meta-analysis was published in F1000Research on 30 October 2015 and revised in January 2016; it carries its open referee reports and data files.

What investigators concluded

Wagenmakers and colleagues argued that Bem's analysis "was partly exploratory", that one-sided tests overstated the evidence, and that re-analysed with a default Bayesian test "the evidence for psi is weak to nonexistent". Ritchie, Wiseman and French repeated Experiment 9, chosen because it had the largest effect, at three universities with 50 participants each, pre-registered through a replication registry Wiseman kept. None of the three, separately or together, was significant (combined p = .83). By French's account in the Guardian, the Journal of Personality and Social Psychology declined to send the paper for review because it "does not publish replications"; Science and Psychological Science did the same, and at the British Journal of Psychology one of the two referees, the one who had reservations, was Bem himself. Live Science reported the journal's refusal in the same terms. Bem's response, as Live Science reported it, was that it was "premature to conclude anything about the replicability" of the effect, and that sceptical experimenters might influence their results.

Galak and colleagues then ran seven experiments on Experiments 8 and 9, with 3,289 participants, many of them online, and found no effect. Their meta-analysis of all replication attempts to that date put the average at d = 0.04, "no different from 0".

The leading explanation: analytical flexibility

The leading conventional explanation is that Bem's results were produced by flexible analysis and selective reporting, not by precognition. The same year, Simmons, Nelson and Simonsohn showed by simulation and experiment how undisclosed choices in data collection and analysis make false positives easy to obtain. Ulrich Schimmack, in Psychological Methods in 2012, used Bem's paper as a worked example: with studies of modest power, a run of nine experiments almost all significant is itself improbable, and suggests that questionable practices helped. Ed Yong's 2012 Nature feature on psychology's replication problem opened its reference list with Bem. The explanation is strong because it needs nothing new and fits the failed pre-registered replications. Its weakness is that it is a diagnosis of method, not a demonstration: no one has shown which choices produced which result in each of the nine experiments.

What remains disputed

Bem has not withdrawn the claim. His 2015 meta-analysis with Patrizio Tressoldi, Thomas Rabeyron and Michael Duggan gathered 90 experiments from 33 laboratories in 14 countries and reported an overall effect of z = 6.40, with a small effect size (Hedges' g) of 0.09; excluding Bem's own studies the effect was 0.06. It included the Ritchie and Galak replications, and it found that Galak's online sessions carried much of their null result. The authors argued that the database was not significantly compromised by selection bias or "p-hacking", though they acknowledge the meta-analysis was not itself pre-registered. Critics read an effect that shrinks towards zero under stricter conditions as the expected signature of bias. The two readings of the same database have not been reconciled; the paper's lasting place is as a case study in the reform of psychology's methods.

Sources

• Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect (archived) — Daryl J. Bem, Journal of Personality and Social Psychology 100(3), 2011 (abstract read via Europe PMC). The nine experiments, participant numbers, effect sizes.
• Why psychologists must change the way they analyze their data: the case of psi (archived) — Wagenmakers, Wetzels, Borsboom and van der Maas, JPSP 100(3), 2011 (abstract). The same-issue critique and Bayesian re-analysis.
• Failing the future: three unsuccessful attempts to replicate Bem's 'retroactive facilitation of recall' effect (archived) — Ritchie, Wiseman and French, PLoS ONE, 2012. The three pre-registered replications, sites and results.
• Correcting the past: failures to replicate psi (archived) — Galak, LeBoeuf, Nelson and Simmons, JPSP 103(6), 2012 (abstract). Seven experiments, N = 3,289, d = 0.04.
• Feeling the future: a meta-analysis of 90 experiments on the anomalous anticipation of random future events (archived) — Bem, Tressoldi, Rabeyron and Duggan, F1000Research, 2015–16. The proponents' meta-analysis, its search, exact and modified replications, and the pre-registration point.
• Precognition studies and the curse of the failed replications — Chris French, The Guardian, 2012. The journals' refusals and Bem as referee.
• Study: Psychic ability doesn't hold up (archived) — Stephanie Pappas, Live Science (via Christian Science Monitor), 2012. The journal's refusal, independently; Bem's response.
• Journal's paper on ESP expected to prompt outrage (archived) — Benedict Carey, New York Times, 2011 (copy hosted by DESY). Advance circulation and the editor's defence of the review.
• The ironic effect of significant results on the credibility of multiple-study articles (archived) — Ulrich Schimmack, Psychological Methods 17(4), 2012 (abstract). Bem's paper as an example of too many significant results.
• False-positive psychology — Simmons, Nelson and Simonsohn, Psychological Science 22, 2011 (abstract). How analytical flexibility inflates false positives.
• Replication studies: Bad copy (archived) — Ed Yong, Nature 485, 2012 (front matter and references only). Bem's paper in the replication debate.