Product planning
Define a useful success check for a product experiment
Write down what you expect to happen, what you will observe, where the data comes from and how long you will watch. Then state which result would lead you to continue and which would lead you to stop, before you look at any numbers.
A text to start with
We are testing [change] because users have trouble with [user problem]. We think [hypothesis]. Data available: [sources and what they record]. The experiment will run for [time window] with [who is included]. Draft a success check with the observation that would support the hypothesis, the data source, the window and a decision rule for continuing, changing or stopping. Do not invent baseline figures.
Example request. Change the details to fit your situation.Experiment success check
Experiment name: [ ] User problem addressed: [ ] Change being tried: [ ] Hypothesis: [ ] Observation that would support it: [ ] Data source and what it records: [ ] Who is included: [ ] Start date and end date: [ ] Other changes during the window: [ ] Continue if: [ ] Change approach if: [ ] Stop if: [ ] Rule agreed by and date: [ ] Outcome observed: [ ] Decision taken: [ ] Later changes to the rule and reason: [ ]
Measures chosen after the fact
The usual mistake is picking the measure after seeing which one moved. Fix the observation and the decision rule in writing first, with a date. Check that the measure relates to the user problem: a rise in clicks does not show that a task became easier. Confirm that the data source records what you think it records and covers the people in the experiment. Note anything else that changes during the window, such as an unrelated release, since it can muddy the result. If the window is too short to show the behavior you care about, say so.
Ask another agent to check the result
Check this experiment plan. Name any measure unrelated to the stated user problem and any metric or threshold that looks chosen after results were seen. Ask whether the data source records the claimed behavior and whether the window is long enough.
Lock the rule before launch
Share the success check with the people who will decide and ask them to challenge the decision rule. Save the agreed version with its date. At the end of the window, apply the rule as written and record the outcome. If you want to change the rule afterward, log the change and the reason separately from the result.