An experiment is not finished until the decision is recorded
Keep hypotheses, observations and decisions together so a future team can understand what a test did and did not establish.
Norla Editorial5 min
A growth team can accumulate many test assets while retaining very little reusable knowledge. The campaign is archived, the spreadsheet stays in a personal folder and the final decision survives as a message in a busy channel. Months later, a similar idea returns because no one can explain what the earlier test actually showed.
Create a record before launch with the problem, hypothesis, variants, measurement definitions and analysis plan. Include dependencies and conditions that would make the result difficult to interpret. This is not an attempt to predict every complication; it establishes what the team intended to learn before seeing the outcome.
After the observation period, record what happened and how the execution differed from the plan. Preserve changes to exclusions, metrics or timing. A result with limitations can still be useful, but its interpretation should reflect those limits. Do not call an observational difference a causal effect simply because it appears beside a test label.
Close the record with a decision and a reason: adopt, revise, investigate further or stop. Name the owner of the next step and what evidence would reopen the question. AI can help organize the notes and identify missing fields; the team remains responsible for the interpretation and the decision that follows.
Put it into practice
Write the hypothesis and analysis plan before launch.
Record execution changes and exceptions.
Keep observations separate from causal claims.
Close with a decision, owner and reopening condition.
Specific questions, practical answers, and the next detail to check. Prepared by Norla Editorial.
Q
Question 01
An experiment ends without enough usable evidence to choose between two options, but the team still needs an operational decision. What should the registry record besides the inconclusive result?
A
Norla Editorial · Answer
Record the decision actually taken, the operational reason for it, and the limits of the available evidence. The team might retain the current setup, defer a change, or plan another study. Keep that practical choice distinct from a conclusion that the experiment proved one option superior, and assign the next review question.
↳
Follow-up question
What if the team chooses the new option because it is easier to maintain, even though the experiment did not establish better campaign performance?
A
Norla Editorial · Clarification
State maintenance simplicity as the decision criterion and preserve the unresolved performance question. That is a legitimate operational rationale if the responsible owner accepts it. The record should not later be summarized as a performance win merely because the team adopted the new option after conducting an experiment.
Q
Question 02
A second team wants to repeat a previously rejected campaign idea, arguing that the audience has changed. Which earlier records would help decide whether the proposal is genuinely new?
A
Norla Editorial · Answer
Compare the original question, audience, intervention, constraints, decision criteria, and reasons for rejection with the new proposal. Identify which assumptions have changed and which remain. The earlier result can inform the plan without automatically settling it, but the new team should explain why the changed context could alter the relevant decision.
↳
Follow-up question
If the earlier rejection was caused by a broken measurement setup rather than the idea itself, how should the new proposal represent that history?
A
Norla Editorial · Clarification
Describe the earlier study as unable to answer the intended question and link the measurement failure. The new proposal should show how that limitation will be addressed before relying on another comparison. Avoid labeling the idea unsuccessful when the record only establishes that the previous evaluation was not usable.
YOUR SIDE OF THE DISCUSSION
Add your perspective.
Your own notes stay private on this device. They are not sent to other members.
Background discussion & source notes
NORLA EDITORIAL / DISCUSSION DESK
Let’s take the question further.
Practical follow-ups, open questions and considered answers from the Norla editorial desk.
5 official discussion notes
Norla Editorial@norla.editorial · Note 01
Preserve the decision, not only the chart
An experiment record should explain what the team decided and why, not merely retain a screenshot of results. Write down the initial question, intervention, comparison, decision rule, and constraints before interpreting the outcome. After the review, record whether to adopt, reject, continue investigating, or make no change. Include the uncertainty that affected that choice. This gives the next team a way to distinguish an inconclusive test from an unsuccessful idea. Without that memory, similar proposals can repeatedly return as new discoveries, while context that justified a previous decision disappears when a report or operator changes.
Consider a campaign test that starts during a landing-page redesign and a tracking update. An observed difference may be difficult to attribute to the intended change alone. Google Ads describes experiments that divide traffic and budget between an original campaign and an experiment; that mechanism does not remove the need to understand other relevant changes in your setup. Maintain an event log alongside the experiment record. For this exercise, identify which changes affect both groups, which may affect only one, and which cannot be assessed from available evidence. Keep those distinctions visible in the final decision brief.
A controlled experiment and an observational comparison can both inform work, but they support different kinds of conclusions. If traffic cannot be allocated appropriately or the intervention cannot be isolated, a descriptive before-and-after review may still identify questions worth investigating. Label it accurately and avoid presenting it as proof of causal impact. When a formal experiment is feasible, define the comparison and review criteria in advance with an appropriately qualified analyst. The choice should reflect the decision, available access, and practical constraints. It should not be driven by a desire to call every dashboard movement an experiment.
A common mistake is to inspect many metrics, find one favorable movement, and present it as the original objective. Exploratory observations can be valuable, but they should be distinguished from the pre-specified decision criteria. Preserve the original plan and add a separate section for new questions raised by the result. If tracking failed or the comparison became unsuitable, record that limitation rather than rescuing the narrative with a convenient metric. The experiment registry should make it possible for another reviewer to reconstruct what was intended before the outcome was known and what changed during the work.
At the end of the review, write a brief containing the decision, supporting observations, unresolved limitations, responsible owner, and next check. Link it to the original plan and the dated source exports. If the decision is to make no change, say whether that reflects insufficient evidence, operational constraints, or an observed result against the acceptance rule. Those are different outcomes. Ask a colleague to read only this brief and explain the decision back to you. Any missing context they identify belongs in the record before the experiment is archived or used to justify a future proposal.
NORLA EDITORIAL / FIELD NOTES
Continue exploring.
Working methods, decisions to document, and useful questions to take into your next project.