Scientific Literature Experiment

How much of the published literature is actually true?

Build a scientific literature

Imagine that scientists have 200 ideas they want to test. Some of these ideas are correct in reality, while the rest are false.

Each hypothesis is tested statistically. If the result is significant, you count this as a publication in the literature. If not, the result is discarded and not published. This simulates a tendency to preferentially publish positive findings.

In reality it is unknown how many hypotheses are true upfront. In this experiment you can choose how many of the 200 ideas are actually correct, and explore what happens when you change the alpha level and statistical power.

Hypotheses to test: 200

Number of correct ideas 20

How many of the 200 hypotheses are actually true in reality?

Alpha 0.05

The probability that a false hypothesis nevertheless produces a significant result.

Statistical power 80%

The probability that a genuinely true hypothesis produces a significant result.

The 200 hypotheses

Each square represents one hypothesis. Significant results enter the published literature. The other results disappear.

True idea, published

False idea, published

True idea, not published

False idea, not published

0 Correct ideas

0 False ideas

0 Published findings

0 Unpublished findings

The published literature

Now throw away all non-significant results. What remains is the literature that readers will see.

No hypotheses were published.

Published and true

Published but false

0% of the published findings came from ideas that were actually true

What would you expect in the long run?

Repeat the simulation a few times while keeping the settings unchanged. The exact outcome changes through chance, but the common situation should become clear.