expectation: A modern take on statistical A/B testing with e-values and martingales

Modern data science demands flexible statistical methods that can handle sequential data analysis and continuous monitoring. Traditional p-values, while widely used, have limitations when dealing with sequential testing scenarios. This talk introduces a Python library that implements e-values and e-processes, offering a more natural approach to measuring statistical evidence and enabling true sequential testing.

Outline:

1. Statistical toolkit

• Current tools
• Purpose and fundamental concepts
• Challenges in modern statistics
• Type 1 error concerns
• Optional stopping problems

1. Sequential testing

• Origins
• The concept of sequential testing
• Peeking

1. e-values

• What are e-values?
• Definitions and concepts
• Betting interpretation
• Wealth process
• Ville's inequality
• Anytime valid inference
• p-value vs. e-value differences

1. Python library

• Architecture
• Core components
• Installation and basic setup

1. Demo 1: A/B testing

2. Beyond A/B testing

• Broader applications
• Conformal e-testing
• Confidence sequences

1. Demo 2: It is a versatile library

2. Acknowledgments

Q&A Session