Once you have conducted an A/B test, it’s time to measure the results and determine the margin of error. The results will show whether the changes you made had the desired impact. The margin of error indicates how confident you can be in the results based on your sample size.
To measure results, compare key metrics like average gift amount, click-through rates, or response rates between your control group and treatment group. The difference between the two groups is your effect size. A larger effect size means the changes you made likely had a bigger impact. However, you need to consider your margin of error before acting on the results.
The margin of error refers to the amount of variability you would expect in results if you repeated your test multiple times. It depends on your confidence level, often 95% or 99%, and your sample size. A larger, more representative sample leads to a smaller margin of error and more confident results. With a small sample, your results could vary more if you rerun the test.
For example, if your test shows a 10% increase in average gift with a margin of error of 5% at 95% confidence, you can be reasonably sure your changes had a positive effect. The effect is larger than the margin of error. But if the increase was only 2% with a 4% margin of error, the effect could be due to chance. You cannot rule out the null hypothesis that there was no real difference.
Nonprofits often have limited resources for testing, but a smaller sample does not mean you cannot gain useful insights.
Aim for at least 200-500 donors or constituents in your test for a representative sample and reasonable margin of error. If possible, partner with other nonprofits to increase your sample size. Consider a consultant if you want to test major changes or conduct multivariate testing. With the right approach, you can measure results, understand your margin of error, and make data-driven decisions to better serve your mission.
View the full recording of this session in our Resource Library.