Why a gut feeling isn't enough
You ship a redesigned checkout button. Sales go up 3% the next week. Was it the button, or was it Tuesday? Traffic mix, seasonality, and plain randomness move your numbers every day — enough that a change with zero real effect can still look like a win if you only glance at before-and-after totals.
An A/B test fixes this by showing both versions at the same time, to randomly split groups of otherwise-identical users, and asking a statistical test whether the gap between them is bigger than chance would produce on its own.
How a test runs end to end
Create an experiment flag with a control and one or more treatment variants. Each user is randomly assigned a variant and stays on it for the life of the test — no one flickers between the old and new experience.
When a user does the thing you're measuring — completes a purchase, clicks the button, finishes onboarding — your backend reports that conversion event tagged with which variant they saw.
A test needs enough people and enough time to average out noise. A result after 20 visitors on each side isn't a result — it's a coin flip you're reading tea leaves into.
The Performance tab runs a two-proportion significance test between each treatment and the control, and reports a p-value, the relative uplift, and a winner badge once the result clears your confidence threshold (95% by default).
Reporting a conversion
From the service that owns the outcome — your checkout flow, your onboarding step — post the event once the user has actually converted:
POST /experiments/events
{
"flagId": "checkout-button-color",
"stage": "prod",
"eventName": "purchase",
"userId": "user-42",
"variantKey": "treatment"
}Three ways to fool yourself
Checking results every hour and stopping the moment it looks significant inflates your false-positive rate — early swings regress to the mean as more data comes in. Decide your sample size or run time up front, then wait for it.
Test five button colors simultaneously and one will look like a "winner" by chance alone, even if none of them do anything. Keep tests focused — one clear hypothesis at a time.
A new design gets extra clicks just for being new. If the effect only shows up in week one and fades by week two, you measured curiosity, not a real improvement.