Commerce OS guide

Experiments

Guide status: Experimentation guidance. Last reviewed 14 August 2026. Creation, allocation, stopping and result states depend on the installed Liva 7 capability and must be verified before use.

Purpose

Experiments supports controlled comparisons when descriptive evidence is not strong enough for a decision. It helps turn a clear hypothesis into an accountable design, protect the customer experience with guardrails, and retain what was learned even when the result is inconclusive.

Not every question needs an experiment. Use one when uncertainty matters, the intervention is safe to test and the store can support a meaningful comparison. Do not use experimentation to bypass product, brand, privacy or merchant approval requirements.

Before you begin

  • State one decision the experiment is intended to inform.
  • Define the current experience and proposed alternative.
  • Confirm eligibility, placement, measurement and technical readiness.
  • Select a primary measure and customer or commercial guardrails.
  • Set the authority, owner and conditions for stopping.

Design the comparison

Write the hypothesis before viewing the result. Define the eligible population, unit of allocation, variants, primary measure, guardrails and planned observation window. Avoid changing these retrospectively to produce a more favourable story.

Check that the variants differ in the intended way and that customers cannot inadvertently receive conflicting experiences. Consider market, device, inventory, promotion and repeat-visit effects. If the installed application does not provide the necessary allocation and analysis controls, treat the proposal as an experiment plan rather than a running experiment.

Run with control

  1. Verify both the control and treatment on the live storefront.
  2. Confirm valid exposure data before interpreting outcomes.
  3. Monitor technical and customer guardrails without repeatedly judging the primary result too early.
  4. Pause when safety, integrity or measurement is compromised.
  5. Complete the planned window unless a pre-defined stopping rule applies.

Interpret the result

A positive directional difference may still be uncertain. A neutral result can rule out a material effect or reveal that the test lacked sensitivity. An inconclusive result is not a failure if the design and limitations are recorded honestly.

Review practical importance alongside statistical or analytical confidence. A small difference may be real but not operationally worthwhile; a promising large difference may require more evidence.

Safe use and fallback

If allocation, exposure tracking, eligibility or variant delivery fails, stop interpretation and preserve the approved default experience. Do not combine compromised data with valid data without a documented method. Customer harm, misleading content or a broken purchase path is a reason to stop immediately, regardless of commercial promise.

You are finished when

  • the original hypothesis and design are retained;
  • integrity and guardrails have been reviewed;
  • the conclusion matches the evidence and includes limitations; and
  • the resulting decision and follow-up are recorded.