Yang Yang: The Replicability of Scientific Findings Using Human and Machine Intelligence

In top journals, more papers fail than pass replication tests and papers failing replications spread as widely as replicating papers. This dynamic raises research costs by over 20bn annually, jeopardizes the literature, and exposes the need for new methods for predicting replicability. Using 96 studies that underwent rigorous manual replication, we developed an artificial intelligence (AI) model that predicts a paper’s replicability. We then tested the model on 317 diverse out-of-sample studies that span disciplines, methods, and topics.

We find that AI predicts replicability better than statistics and individual reviewers and as accurately as prediction markets, the gold standard of replicability methods. Further, AI generalizes to out-of-sample data at AUC levels up to 0.78. Finally, tests indicate that the AI model does not show biases common to human reviewers. We discuss how AI can address replication problems at scale in ways that current methods cannot and can advance research by combining human and machine intelligence.
  

Yang Yang

Research Assistant Professor, Kellogg School of Management, Northwestern University

I am a Research Assistant Professor at the Kellogg School of Management, Northwestern University. My principal research interest lies in the area of data mining/machine learning, computational social science and science of science. I study the link between the social network and leadership attainment and how social network can help women to achieve placement in leadership. Based on organizational theory, I develop a model for estimating a terror group’s future lethality by inferring from latent variables its hidden capabilities and resources. This model has an unique early warning signals. I also test the ability of artificial intelligence to address the replication problem in science. The goal is to demonstrate how AI can address replication problems at scale in ways that current methods cannot and can advance research by combining human and machine intelligence.