Our testing approach

The problem with traditional A/B testing

Standard A/B testing tools use frequentist statistics — specifically, the two-proportion Z-test. This approach asks: “If there were no real difference between variants, how likely is it that we would see a difference this large by chance?” When the probability drops below 5%, we declare a winner.

This works well when you have thousands of visitors per variant. But it was never designed for the reality most SMB marketers face: a landing page receiving 200–500 visitors a week, where waiting for statistical significance means weeks or months of uncertainty — by which time the campaign, the offer, and the season may all have changed.

Worse, the binary yes/no output of traditional testing is misleading at low sample sizes. “Not significant” does not mean “no difference exists” — it means “we do not have enough data to be sure.” Those are very different things, and conflating them leads to bad decisions.

The Bayesian alternative

SplitCheck uses a Beta-Binomial Bayesian model to analyse your test results. Instead of a binary significant/not-significant output, we answer a more useful question:

“Given the data we have collected so far, what is the probability that Variant B outperforms Variant A?”

This probability — displayed prominently as the large percentage in your Results panel — updates continuously as visitors arrive. At 20 visitors per variant, it gives you a genuine directional signal. At 200, it is reliable enough to act on.

We also report a 90% credible interval on the expected lift. This tells you not just the direction of the effect, but a realistic range for its size. A credible interval of +4% to +21% means we are 90% confident the true lift falls somewhere in that range — a far more actionable output than a p-value.

How the model works

Each variant's conversion rate is modelled as a Beta distribution — a probability distribution over probabilities. We start with a non-informative prior (Beta(1,1) — no assumptions about likely conversion rates) and update it as data arrives.

After each new visitor and conversion, the posterior distributions for A and B are updated. We then use Monte Carlo simulation (10,000 draws) to estimate the probability that B's true conversion rate exceeds A's — this is the P(B beats A) figure you see in your dashboard.

The decision threshold — defaulting to 95% — means we recommend declaring a winner when we are at least 95% confident in the direction. You can adjust this in your test settings based on the stakes involved.

Parallel validation

We run the traditional frequentist Z-test in parallel with the Bayesian engine on every test. The frequentist result is available under “How does this compare to traditional A/B testing?” in your Results panel.

This parallel approach serves two purposes: it lets you cross-reference results if you prefer the traditional framework, and it feeds our ongoing research into the comparative performance of both methods at SMB traffic volumes. As we accumulate test outcomes across our customer base, we are building an empirical dataset that will allow us to further refine and validate the Bayesian approach — and eventually develop industry-specific priors that make results even faster to reach.

Data quality indicators

Because we report results at lower sample sizes than traditional tools, we believe it is important to be honest about data quality. Every test result includes a data quality indicator:

We deliberately show low-quality results rather than hiding them, because even at low sample sizes, a 92% probability signal is meaningful context — you just should not bet your entire campaign budget on it yet.

What we do not do

We do not use peeking corrections, sequential testing adjustments, or multi-armed bandit algorithms. These are legitimate approaches, but they add complexity that can obscure what is actually happening in your test. Our goal is interpretability: a single probability you can explain to a client or a CMO in one sentence.

We also do not inflate results or declare winners early to make the product look impressive. If the data says “continue,” we say “continue.” An honest 63% is not as exciting as a false 97%, but it is the number that helps you make better decisions.

Built on academic foundations

SplitCheck's statistical engine is grounded in the academic literature on Bayesian approaches to online experimentation. Our ongoing research focuses specifically on the underserved problem of valid causal inference in low-traffic SMB marketing contexts — an area where the existing literature has significant gaps.

We are actively developing research papers on this methodology and welcome collaboration with academic institutions and industry researchers. If you are working in this space, we would love to hear from you.