What 162 Vinted A/B Tests Reveal About Your Conversion Metrics

This article analyzes 162 A/B tests from Vinted, Europe’s largest secondhand clothing marketplace, to compare safe t-tests, classical t-tests, mSPRT, and safe proportion tests. Results reveal that while safe t-tests often align with classical tests, they excel on short-term metrics like searches and sessions but underperform on long-term outcomes like transactions. The safe proportion test also outperforms the χ² test in certain SRM detection scenarios, offering practical insights for experiment design in fast-moving online marketplaces.