A reusable framework for analyzing controlled experiments using frequentist, Bayesian, and sequential methods, with multiple comparison correction and power analysis.
$ pip install -r requirements.txt && streamlit run app.py
Key results
Three controlled experiments analyzed with dual frequentist and Bayesian frameworks
3
Experiments analyzed
p<.001
Email subject line test
0.996
P(B > A), website redesign
2 / 3
Significant experiments
Methodology
Three experiments (website redesign, pricing change, email subject line) with conversion and revenue metrics. The frequentist path uses z-tests for proportions and Welch's t-tests for continuous outcomes, with Cohen's h and d effect sizes and multiple comparison correction via Bonferroni, Holm, and Benjamini-Hochberg FDR. The Bayesian path uses a Beta-Binomial conjugate model with uninformative priors and 100K posterior samples. Sequential monitoring applies O'Brien-Fleming spending functions for valid early stopping.