What OCE Datasets Teach Us About Statistical Testing in A/B Experiments

The article examines the use of OCE datasets as a benchmark for comparing statistical methods in online controlled experiments. It focuses on how the safe t-test performs against the classical t-test and mSPRT, revealing that while it detects more effects, results can be skewed by novelty effects. This insight is crucial for practitioners implementing anytime-valid testing, as early rejections may not reflect a feature’s true long-term impact.