Glossary
Incrementality Testing
Incrementality testing is an experimental method that measures the causal lift a marketing channel or campaign produces by comparing conversion rates between a randomly assigned treatment group exposed to the marketing and a holdout group that is withheld from it. The difference in conversion rates between the two groups is the incremental effect: the portion of conversions that would not have occurred without the marketing.
Why incrementality testing is different from attribution
Attribution, whether single-touch or multi-touch, is an observational method. It looks at historical journeys and distributes credit based on correlation: this touchpoint appeared before conversions, so it gets credit. Correlation does not equal causation. A branded search click correlates almost perfectly with conversion, but in many cases the buyer had already decided to purchase before the search. The search was a navigation step, not a persuasion step. Attributing it as persuasion overstates its causal value.
Incrementality testing introduces random assignment, which breaks the correlation between intent and exposure. By withholding a channel from a randomly selected holdout group, you can observe whether those buyers convert at a lower rate than the exposed group. If they do, the channel is producing genuine incremental conversions. If they convert at the same rate, the channel is capturing conversions that would have occurred organically, and the attributed credit is misleading.
Types of incrementality tests
Geo holdout tests
A geographic region is selected as a holdout market and marketing spend is withheld from that region while running normally in comparable regions. Conversion rate differences are attributed to the paused spend.
Audience holdout tests
Ad platforms assign a random percentage of the target audience to a holdout group that does not see the ads. The platform measures conversion lift in the exposed group vs. the holdout.
Time-based tests
Spend is paused during one time period and run normally during comparable periods. Seasonal adjustments and external factors must be controlled carefully to isolate the marketing effect.
Matched market tests
Statistical methods match test and control markets on pre-period metrics before the experiment, improving the reliability of post-period comparisons in geo experiments.
Incrementality testing vs. attribution: the relationship
Incrementality testing and marketing attribution serve different functions in a measurement stack. Attribution provides continuous, granular measurement at the campaign and creative level: it tells you which touchpoints appeared in buyer journeys and how credit is distributed under your chosen model. Incrementality tests provide periodic causal validation: they tell you whether a given channel is actually driving conversions that would not have happened otherwise.
The standard practice is to run attribution continuously for operational optimization and run incrementality tests periodically to calibrate the attribution model. When an incrementality test shows that a channel's causal lift is substantially lower than its attributed credit, the attribution model is over-crediting that channel and budget decisions based on it are likely flawed.
A similar relationship exists between incrementality testing and marketing mix modeling. MMM estimates channel contribution from aggregate data; incrementality tests validate those estimates experimentally. Running both and triangulating the results produces more reliable budget decisions than either method alone.
Practical considerations for B2B incrementality testing
B2B incrementality testing is harder than B2C for two reasons. First, conversion volumes are lower, which means tests require longer run times to achieve statistical significance. A paid social holdout test that would reach significance in two weeks for an e-commerce brand might require six months for a SaaS company with a 3% demo-to-close rate. Second, the conversion event for B2B is typically not a purchase but an MQL, SQL, demo request, or pipeline opportunity, each of which has a different relationship to final revenue.
For teams using data-driven attribution, incrementality tests are particularly valuable as calibration inputs: they provide ground-truth causal estimates that can be used to adjust the statistical weights the DDA model assigns to individual channels.
Incrementality testing in AttriByte
AttriByte's measurement framework is designed to complement incrementality testing. The platform's warehouse-native architecture makes it straightforward to define holdout cohorts in your data warehouse and compare their conversion rates against exposed cohorts using the same identity-resolved event data that feeds the attribution models. Atlas AI can flag channels where attribution credit is disproportionately high relative to their observed incremental lift, surfacing candidates for formal holdout testing.
Related glossary terms
Validate your attribution with real experiments
AttriByte's warehouse-native data model makes holdout cohort analysis straightforward alongside continuous attribution measurement.