Definition
The p value is the probability, assuming the null hypothesis is true, of observing data at least as incompatible with the null hypothesis as the data actually obtained.
Interpretation
A small p value indicates that the observed data would be unusual if the null hypothesis were true. It does not measure the probability that the null hypothesis itself is true or false.
Statistical significance
A threshold such as p < 0.05 is commonly used, but the threshold should be prespecified. Statistical significance does not automatically imply clinical importance.
Common errors
A p value does not tell us:
- the size of an effect
- the clinical importance of an effect
- the probability that the result occurred “by chance” in a simple sense
- the probability that the alternative hypothesis is true
Relationship to confidence intervals
Confidence intervals provide information about both effect size and precision and should usually be interpreted alongside the p value.
Type I and Type II error
- Type I error: rejecting a true null hypothesis
- Type II error: failing to reject a false null hypothesis
Diagram
