Charta is on Google Play: practise on the go. Get the app
Charta

Explain what a p-value means and the two kinds of testing error

CFA® Level I · Quantitative Methods · Estimation and Hypothesis Testing · QM.7.b

The idea

A p-value is the probability of observing a result at least as extreme as the one obtained, if the null hypothesis were true. It is not the probability that the null is true, and it says nothing about how large or important an effect is. A Type I error rejects a true null, and its probability is the significance level the analyst chooses. A Type II error fails to reject a false one. Tightening the significance level reduces Type I errors and increases Type II errors, so the choice is a judgement about which mistake costs more, not a technical default.

A practice question

From the question bank, with the reason each wrong option is wrong.

What is a Type I error?

  1. Rejecting a null hypothesis that is actually true — correct
  2. Failing to reject a null hypothesis that is actually falseWhy not: That is a Type II error — the false negative. A Type I error is the false positive.
  3. Choosing the wrong test statisticWhy not: That is a specification mistake, not an inference error. Both error types assume the test was correctly specified and concern which way the conclusion went.

A Type I error is a false positive, and its probability is the significance level the analyst sets in advance. A Type II error is the opposite mistake.

A harder one

A test returns a p-value of 0.03. What does this mean?

  1. There is a 3% probability the null hypothesis is trueWhy not: The misreading that causes most of the trouble. The p-value is computed *assuming* the null holds, so it cannot also be a probability about the null.
  2. If the null were true, a result this extreme would occur about 3% of the time — correct
  3. The effect is large enough to matter economicallyWhy not: It says nothing about size. A tiny, worthless effect measured on a large enough sample produces a small p-value.

The p-value is computed assuming the null holds, so it describes the data's extremeness under that assumption — not the probability of the hypothesis itself, and not the size of the effect.

The hardest tier

An analyst lowers the significance level from 5% to 1%. What is the trade-off?

  1. Both error types fall togetherWhy not: Only a larger sample reduces both at once. At a fixed sample size the two move in opposite directions.
  2. The test becomes invalid for small samplesWhy not: Validity depends on the assumptions of the test and the distribution used, not on the threshold chosen.
  3. Fewer false positives, but more false negatives for a given sample size — correct

Demanding stronger evidence reduces Type I errors and raises Type II errors. Only a larger sample improves both at once, which is why power calculations matter more than the choice of threshold.

In the app: 9 practice questions on this outcome, a narrated walkthrough with diagrams, spaced review that brings it back before you forget it.

Take the free diagnostic →

More in Estimation and Hypothesis Testing