Experimentation

A conversion win that lowers margin is not a win. The most common failure in growth experimentation is optimising a single funnel metric while a cost line moves the wrong way underneath it. Agents make this easy to avoid, because they can hold several objectives at once.

TL;DR in 60 seconds

• Optimising conversion alone can drop average order value, raise CAC, or pull in worse-fit accounts.
• Give every experiment a primary metric and a guardrail metric before it ships.
• Let agents score variants against revenue and margin together, not clicks.
• Kill tests that win on the surface but lose on the guardrail.

Why single-metric optimisation misleads

Say an agent finds a discount banner that lifts checkout conversion 14%. Shipped naively, it looks like a win. But if it drops average order value 20% and pulls in customers who churn faster, the business is worse off. The dashboard turns green while margin turns red.

The fix: paired metrics

Every experiment gets two numbers up front:

1. Primary: the thing you are trying to move (trial-to-paid, booked revenue, activation).
2. Guardrail: the thing that must not degrade (margin, CAC, retention, refund rate).

An agent can measure both against the joined data layer and only recommend shipping when the primary improves without breaching the guardrail. This is where "fine-tuned to your KPIs" earns its keep: the objective function includes margin, so the agent never optimises it away.

What this looks like in practice

• Pricing tests scored on revenue per visitor, not conversion rate.
• Paid experiments scored on contribution margin per acquired customer, not CPA alone.
• Lifecycle sends scored on incremental revenue, not open rate.

Point the agents at the right objective, keep a human on the ship decision, and the tests that survive make the business healthier, not just busier.