Lunchlezing Booking.com

The Statistics Behind Experimentation at Booking.com

by Raphael Lopez Kaufman At Booking.com we have been using online controlled experiments, also called A/B tests, for more than ten years to conduct evidence based product development. Overall, on a daily basis, all members of our departments run and analyse more than a thousand concurrent experiments to quickly validate new ideas. These experiments run across all our products, from mobile apps and tools used by hoteliers to customer service phone lines and internal systems. In this talk, I will try to motivate why online giants such as Google, Facebook, Amazon or Booking.com run online controlled experiments and why we need statistics to analyse experiment results. After having described the theoretical framework which underpins online experimentation, the Neymar-Rubin causal model, I will go into more details about how this framework is used at Booking.com, with concrete examples. Finally, I will describe the common pitfalls we need to guard against to make sure product teams are not chasing statistical ghosts, and I will conclude with some of the statistical challenges we are facing to enable faster and more reliable experimentation across the board. Note: After subscription, showing up is mandatory!