Module 8 Lesson: Hypothesis Testing and p-values
Classroom Explanation
Imagine we are in class and someone puts a small table on the board. The table looks simple, so it is tempting to jump straight to the answer. In statistics, we slow down first. We ask what question the table can answer, what question it cannot answer, and what kind of claim would be safe.
In this module, the goal is to interpret p-values as evidence checks, not proof. The important part is not only the calculation. The important part is the thinking before and after the calculation.
Why This Matters
The common mistake is saying p = 0.041 proves the reminder works. It does not prove the cause, size, or usefulness of the effect. This mistake can make a report sound more confident than the evidence deserves.
Key Ideas
1. A null hypothesis is the default no-difference or no-effect position.
The null hypothesis is the starting assumption for the test. Usually it says there is no difference, no effect, or no relationship worth detecting.
2. A p-value asks how surprising the result would be if the null idea were true.
A p-value is part of an evidence check. It is not a certificate that a claim is true, and it does not tell us whether the effect is important.
3. Statistical significance is not the same as practical importance.
A statistically significant result may still be too small to matter. A practical decision also needs size, cost, risk, and context.
Worked Example
A reminder increases completion by 2 percentage points with p = 0.041. That can be statistically notable but still too small to matter in practice.
Here is the dataset used in this module.
| study_id | question | p_value | effect_size_note | practical_importance | caution |
|---|---|---|---|---|---|
| HT01 | Did the new reminder increase worksheet completion? | 0.041 | Completion rose by 2 percentage points | low | Statistical significance may not be practically useful. |
| HT02 | Did Group A score higher than Group B? | 0.180 | Mean difference was 3 points | unclear | Sample evidence is weak; do not claim a real difference. |
| HT03 | Did the new lesson layout reduce reading time? | 0.004 | Average time fell by 40 seconds | medium | Check whether faster reading harmed understanding. |
| HT04 | Did one of ten tested messages perform best? | 0.049 | Winner beat second place by 1 percentage point | low | Multiple testing makes the result fragile. |
| HT05 | Did practice quizzes improve final review scores? | 0.030 | Mean score rose by 8 points | medium | Check assignment process before using causal language. |
Read the table slowly. First name the unit. Then name the variables. Then name the comparison or pattern. Only after that should you write the conclusion.
How To Think Through It
- State the question in one sentence.
- Name the unit of analysis.
- Identify the outcome and comparison.
- Check the denominator, sample, uncertainty, or assumption.
- Write only what the data supports.
- Add one limit so the result does not overclaim.
Common Mistake
The common mistake is saying p = 0.041 proves the reminder works. It does not prove the cause, size, or usefulness of the effect.
How To Write The Result
The test result is statistically notable, but the observed 2-point increase may be too small to justify a change without cost and context.
Practice Prompt
For each scenario, write the p-value meaning, the practical-size note, and one caution.
Takeaway
When a statistical result feels obvious, pause and ask: what exactly was measured, compared, and assumed?