Model¶

The main point of A/B tests is to find how one version is better than another. The most intuitive approach is to find conversion rates for each version, then compare them. We are not really interested in conversion rates, but their difference. The output of the model is approximation of the random variable $Diff = Rate_B - Rate_A$. If we know it's distribution and cumulative distribution function we can find probability $P(Rate_B < Rate_A)$ or equivalently $P(Rate_B - Rate_A < 0) = P(Diff < 0) = CDF_{Diff}(0)$

$$ \begin{align} Rate_A \sim \text{Uniform}(0,1) \\ Rate_B \sim \text{Uniform}(0,1) \\ Conversions_A \sim \text{Binomial}(Rate_A, N_A) \\ Conversions_B \sim \text{Binomial}(Rate_B, N_B) \\ Diff = Rate_B - Rate_A \end{align} $$