A/B test calculator for UX metrics
Compare two designs, versions or products. Paste the scores for each, and get the difference, its confidence interval, a p-value and the effect size.
Free and private: everything is calculated in your browser and nothing you paste is sent anywhere.
How it is calculated
- Different people in each group: Welch’s t-test, which does not assume equal variances. The confidence interval is for the difference between the two means.
- Same people trying both designs: a paired t-test on each participant’s difference. Values must be in the same participant order in both lists.
- Completion rates: the N − 1 two-proportion test, which Sauro and Lewis recommend for small samples, with an adjusted-Wald interval for the difference.
- Effect size is Cohen’s d: about 0.2 is small, 0.5 medium and 0.8 large (Cohen, 1988). For paired data it is the mean difference divided by the SD of the differences (d_z).
Frequently asked questions
How do I compare two SUS scores?
Paste the individual SUS scores for each design into the two boxes. If different people used each design, choose "Different people" (Welch’s t-test). If the same people tried both, choose "Same people" (paired t-test). The result shows whether the difference is statistically significant and how big it is likely to be.
What does the p-value mean?
The p-value is the probability of seeing a difference at least this large if the two designs were really the same. Below 0.05 is the usual threshold for calling a difference statistically significant, but the confidence interval is more informative because it shows how big the difference plausibly is.
Can I compare completion rates with this?
Yes. Choose "Completion rates" and enter the number of successes and attempts for each design. It uses the N − 1 two-proportion test, which works well even with small samples.
Which test should I use for within-subjects data?
If each participant used both designs, use the paired option. It is more sensitive because it removes differences between people, so it often finds effects that an independent-groups test would miss.