Customer.io announced on November 9 that its Experiments feature can test entire paths through a marketing automation. Customer.io is a messaging platform that lets publishers send emails based on what readers do. A publisher that uses it for onboarding can now compare two sequences of emails and delays inside the same workflow. The feature applies to automations, such as a series that starts when a subscriber signs up. It is unavailable for one-off newsletter broadcasts.
For a publisher that sells subscriptions, the choice might be whether to offer a paid plan in the welcome email or after sending a sample issue. Both paths can be judged on the share of assigned readers who actually pay within a set period. That requires deciding which readers can enter, what counts as payment and how long each reader has to convert.
Jump to a section
Define the test before the first reader enters
Keep the current welcome series as the reference path. In the other path, change when the paid offer appears. Use the same price, audience rules and signup trigger in both paths if the question is whether waiting helps. If the new path also changes the price and the number of messages, you can compare the two packages, but the test cannot isolate timing.
Customer.io’s experiment guide explains how to set a success event and the window in which it counts. For a paid-subscription test, check that your billing or membership system can send a reliable first-payment event to Customer.io. Try it with test profiles first. A subscriber who clicks through to checkout and leaves should produce a different record from one who pays. Decide how refunds and repeat payments will appear in your own analysis. Use the event your system actually records, and compare a few test records against the billing ledger before choosing a winner.
The same guide describes an option to keep a person on the same path across multiple journeys. A repeat signup should still have a defined treatment in your analysis. For example, if a reader signs up twice, count one eligible reader and use the first assignment as the start of the observation period. Check that your integration identifies both signups as the same person before relying on that rule.
How a dashboard snapshot can reverse the result
Imagine a snapshot with 5,000 assigned readers in each path. Because the groups received their signups on different days, 4,000 readers in the reference path and 2,000 readers in the other path have completed a 14-day observation window. Among those mature readers, the first path has 80 payments and the second has 48. These are invented figures for an audit of an exported snapshot. They do not describe how Customer.io calculates its own experiment report.
If you divide those completed-window payments by everyone assigned, the reference path appears to win, 1.60% to 0.96%. The numerator and denominator cover different groups. The paid rate within each mature group is 2.00% and 2.40%, respectively. The direction flips, and that still leaves a question about who entered earlier. If the first completed readers came disproportionately from different campaigns, even a mature-only comparison could reflect that mix. Wait for the planned cohort and check signup dates and sources in both paths. Our guide to tracking signup and paid conversion separately explains how those events can tell different stories.
An imbalance this large should also prompt a check of the export itself. Compare assignment counts by calendar day and signup source. Look for a paused branch, an export with different cutoff dates, or a change in who could enter the experiment.
| Measure | Reference | New path |
|---|---|---|
| Assigned readers | 5,000 | 5,000 |
| Full 14 days elapsed | 4,000 | 2,000 |
| First payments in mature group | 80 | 48 |
| Completed payments ÷ all assigned readers | 1.60% | 0.96% |
| Completed payments ÷ readers with 14 full days | 2.00% | 2.40% |
Choose a winner only if the change earns its place
Customer.io describes three ways to pick a winner, including manual selection and automatic selection at a planned sample or when a path moves sufficiently ahead by the product’s criteria. If a paid offer has a delayed effect or the business needs to check refunds and reader complaints before rolling it out, use the manual choice. Review the result after the last included reader has had the full observation period. The conversion rate comparison calculator can help check the size and uncertainty of the difference once you have comparable counts.
Set the minimum gain you would act on before you start. If a completed test supports a 0.4 percentage point lift and your publication assigns 5,000 similar new readers per month, that would mean about 20 additional paid starts per month. At $30 in expected contribution per start, that is $600 before any additional work or costs specific to the new sequence. Those are planning inputs. Check whether the lift holds after early refunds and whether the longer series increases unsubscribes. A sequence that earns more initial payments can still leave the publication worse off if enough readers cancel or stop receiving the newsletter.