Experiment plan

Plan an experiment - the hypothesis, the test, the metrics and the decision.

ProductMarketing


Owner

@mention the owner

Start

End

Result

⚪ Running | 🟢 Win | 🔴 Loss | 🟡 No difference

Hypothesis

Fill in: we believe that this change, for these users, will cause this result. We will know that it is true when this metric changes by this amount.

Test design

Group

What they see

Size

A (control)

The sign-up page as it is today

50% of new visitors

B, the group that sees the change

What changes for this group

The share of users, or the number

Metrics

Metric

Kind

Target

What you measure

Primary (decides the result), secondary, or guardrail (must not get worse)

The change that you need to see, or the limit for a guardrail

Results

The value of each metric in each group, and whether the difference is statistically significant.

Analysis: link to the dashboard or the analysis, or delete this line

Decision and next steps

What you decide: release the change, remove it or run a new test. Say why.

  • Action · @owner ·

allanswered knowledge management