The experimentation program that actually survives Q3

Experimentation programs follow a predictable arc. Q1: set up, first tests, a couple of clear wins, everyone is delighted. Q2: more tests, smaller effects, some inconclusive. Q3: the wins have dried up, someone asks what the program has delivered this quarter, and the honest answer is "mostly flat results". Q4: budget moves somewhere with a better story.

This isn't a failure of execution. It's the natural consequence of how most programs are designed — and it's avoidable if you change three things: what you test, how you count, and how you report.

Why the wins run out

Early tests win because they fix things that were plainly broken. A confusing form, a missing trust signal, a checkout step that shouldn't exist. These are not really experiments — they're repairs, and repairs have large effects.

Once the obvious defects are gone, you're testing genuine alternatives, where effects are small and the honest answer is often "no meaningful difference". Nothing has gone wrong. But if the program was sold on the early numbers, expectations are now calibrated to a rate that was never sustainable.

The first quarter of a CRO program is debt collection. The rest is investing. They don't pay out at the same rate, and pretending otherwise is what kills the program.

Change one: test the system, not the surface

Programs die on button colours. Not because micro-tests are wrong, but because their ceiling is low — and a portfolio of low-ceiling tests produces a portfolio of small results.

The tests that keep programs alive change something structural:

• Offer and pricing structure — what's included, how it's packaged, how commitment is framed. Consistently the highest-leverage tests available, and consistently the least tested because they need buy-in outside marketing.
• Qualification and routing — who gets asked what, and who reaches a human. Often worth more than any conversion-rate change, because it moves quality rather than volume.
• The step nobody looks at — the confirmation page, the follow-up email, the second visit. Under-tested because they're unglamorous.
• Audience-level differences — the same page can win for one segment and lose for another, and the blended result hides both.

Structural tests are harder to ship, which is exactly why they still have room in them.

Change two: count honestly

Most programs quietly inflate their own results, then lose credibility when finance reconciles.

Stop declaring winners early

Peeking at a test daily and stopping when significance appears manufactures winners. Fix your sample size and duration up front, and hold to it. A test that needs six weeks needs six weeks.

Run tests that can actually conclude

If your traffic can't detect the effect you're looking for inside a reasonable window, don't run the test — you'll get noise and read it as signal. Calculate this before building, not after. Low-traffic sites should test bigger changes precisely because small ones are undetectable.

Count flat results as results

"This didn't matter" is genuinely valuable: it stops the debate, frees the roadmap, and prevents the same argument recurring every quarter. Programs that only report wins train everyone to expect a win rate no honest program can maintain.

Report contribution, not conversion rate

A 6% lift in form starts means little to the person holding the budget. Contribution margin does. This is the same discipline as margin-aware ad measurement — measure in the unit the business actually cares about.

Change three: report so the program survives budget season

Programs are cut in meetings the CRO lead isn't in. What survives is a document someone else can defend on your behalf.

Ours has four parts, updated quarterly:

1. Cumulative contribution from shipped winners, with the assumptions stated plainly. Conservative, so it survives scrutiny.
2. Decisions closed by inconclusive tests — arguments settled, roadmap items removed. This is where the flat results become an asset.
3. What we now know about the audience that we didn't a quarter ago. Insight compounds even when tests don't.
4. The next three structural tests and what they're worth if they work.

The third point is the one that saves programs. A test that fails still buys knowledge, and a program that accumulates knowledge is an asset even in a flat quarter. Say it explicitly or nobody will infer it.

The rhythm

Cadence matters more than volume. What works, sustainably:

• One structural test running at all times, properly powered. Not five underpowered ones.
• A standing queue of small fixes shipped without testing when the change is obviously correct. Don't spend statistical power proving that a broken link should be fixed.
• A monthly readout, thirty minutes, same format every time.
• A quarterly reset where you re-pick the constraint. The bottleneck moves; programs that keep optimising last quarter's bottleneck go flat.

This depends on being able to ship without an engineering queue — which is why the composable content setup tends to be a prerequisite rather than a nice-to-have.

What good looks like at 18 months

Not a rising win rate — that's the arc that kills programs. Instead: a stable cadence of properly powered tests, a documented body of knowledge about your audience, a roadmap picked from evidence rather than opinion, and a quarterly number finance recognises.

Win rates around 20–30% are normal and healthy for a mature program testing real alternatives. If yours is much higher, you're either still collecting debt or stopping tests early.

The short version

Easy wins are repairs, and they run out. Move to structural tests with real ceilings, fix your maths before you start, count flat results as knowledge, and report in contribution. Programs don't die from bad tests — they die from expectations set by the first quarter and reporting nobody can defend.