Module 8 Lesson: Hypothesis Testing and p-values
Classroom Explanation
A p-value answers one question: if there were no effect, how often would data at least this extreme appear? It is computed by assuming no effect, which is exactly why it cannot be the probability that there is no effect.
In this module, the goal is to interpret p-values as evidence checks, not proof. The important part is not only the calculation. The important part is the thinking before and after the calculation.
Why This Matters
The common mistake is saying p = 0.041 proves the reminder works. It does not prove the cause, size, or usefulness of the effect. This mistake can make a report sound more confident than the evidence deserves.
Key Ideas
1. A null hypothesis is the default no-difference or no-effect position.
The null hypothesis is the starting assumption for the test. Usually it says there is no difference, no effect, or no relationship worth detecting.
2. A p-value asks how surprising the result would be if the null idea were true.
A p-value is part of an evidence check. It is not a certificate that a claim is true, and it does not tell us whether the effect is important.
3. Statistical significance is not the same as practical importance.
A statistically significant result may still be too small to matter. A practical decision also needs size, cost, risk, and context.
Worked Example
A reminder increases completion by 2 percentage points with p = 0.041. That can be statistically notable but still too small to matter in practice.
Here is the dataset used in this module.
| study_id | question | p_value | effect_size_note | practical_importance | caution |
|---|---|---|---|---|---|
| HT01 | Did the new reminder increase worksheet completion? | 0.041 | Completion rose by 2 percentage points | low | Statistical significance may not be practically useful. |
| HT02 | Did Group A score higher than Group B? | 0.180 | Mean difference was 3 points | unclear | Sample evidence is weak; do not claim a real difference. |
| HT03 | Did the new lesson layout reduce reading time? | 0.004 | Average time fell by 40 seconds | medium | Check whether faster reading harmed understanding. |
| HT04 | Did one of ten tested messages perform best? | 0.049 | Winner beat second place by 1 percentage point | low | Multiple testing makes the result fragile. |
| HT05 | Did practice quizzes improve final review scores? | 0.030 | Mean score rose by 8 points | medium | Check assignment process before using causal language. |
Ask how many comparisons were made before reading any single p-value. At the conventional threshold, one comparison in twenty comes back significant when nothing is going on, so one significant result out of twelve is what chance produces rather than what a finding looks like.
How To Think Through It
- State the hypothesis being tested.
- Report the p-value with its correct meaning.
- Say how many comparisons were made.
- Report the effect size and its interval.
- Write the conclusion the evidence supports.
Common Mistake
The common mistake is saying p = 0.041 proves the reminder works. It does not prove the cause, size, or usefulness of the effect.
How To Write The Result
The test result is statistically notable, but the observed 2-point increase may be too small to justify a change without cost and context.
Practice Prompt
For each scenario, write the p-value meaning, the practical-size note, and one caution.
Takeaway
A p-value measures evidence against no effect. It says nothing about how large an effect is.