Instrument 6 · the p-value

The number that is not a probability

A p-value of 0.03 does not mean a 3% chance the result is fake, or a 97% chance it is real. It is the chance of seeing data this extreme if nothing were going on — and it shrinks when you collect more data, whether or not anything is there.

Nothing going on, versus something

The same tiny effect, with more data

if nothing were going on if the effect were real what you observed the tail beyond it

The same statement, with the numbers in it

p = 2 (1 - Φ (\frac{| \bar{x} |}{σ / \sqrt{n}}))

0.03 = 2 (1 - Φ (\frac{2.17}{1}))

The denominator is the standard error, σ/√n — and that is the whole problem. Make n big enough and z grows without anything changing. A large p-value is evidence of nothing; a small one is evidence of something, not of something big.

Fine tune

The size of the effect, in σ 0.20

Samples 30

Things you tested 1

Pure noise? no