First, the key terms.

Null hypothesis
A specified reference claim or model, such as no difference in a defined mean.
Type I error
Rejecting the null when it is true under the test’s framework.
Power
Probability of detecting a specified alternative under stated assumptions.

The idea, explained

A p-value is calculated under the null hypothesis and relevant assumptions; it is not the probability that the null is true. A small p-value does not measure effect size or business importance. Tests can make false-positive and false-negative errors, and repeated comparisons or stopping as soon as a result looks favorable can distort interpretation. A good experiment specifies its objective, assignment unit, outcome, sample plan and analysis before inspecting results. Ethical and operational constraints also affect whether experimentation is appropriate.

STEP 1

Define the decision and effect

Decide which outcome matters and what magnitude would justify a change. Conversion rate, processing time and complaints can move differently. Preselect the primary measure and relevant guardrails so a favorable metric is not chosen after the fact.

STEP 2

Assign and analyze correctly

Random assignment helps balance confounders on average. Users, classrooms and transactions are different assignment units; interference or repeated measures can break simplistic independence assumptions. Confirm exposure and data quality before interpreting a difference.

STEP 3

Report more than a threshold

Give group counts, effect estimates, intervals, limitations and practical costs. Failure to reject does not prove equality. A statistically detectable tiny gain may not offset deployment, maintenance or user harm.

Follow a worked example.

A hypothetical signup experiment has 1,000 users in each group. A gets 100 signups and B gets 120. Observed rates are 10 percent and 12 percent: a 2 percentage-point increase and a 20 percent relative increase.

Walk it through, one step at a time

  1. Keep absolute and relative changes distinct. Neither number alone establishes statistical uncertainty.
  2. Use the planned analysis appropriate to assignment and sample size, and report an interval rather than inferring significance from the raw difference alone.
  3. Check guardrails such as accidental signups or unsubscribe rates. More signups are not necessarily better if consent quality declines.

Where people get stuck

Do not keep trying outcomes, subgroups and stopping dates until one produces a desirable p-value.

Now try it yourself.

Pause before opening each answer. Explain your reasoning in your own words.

1. Check the core idea

Does a p-value of 0.03 mean there is a 97 percent probability the tested treatment works?

Reveal the answer & reasoning

No. That interpretation reverses the conditional statement and ignores the model and assumptions. Effect estimation and decision relevance need separate reasoning.

2. Apply what you learned

A huge sample detects a one-second time saving that costs more to implement than it saves. Must it be launched?

Reveal the answer & reasoning

No. Statistical evidence of a difference and a beneficial decision are different questions. Consider magnitude, costs, risks and affected users.