Speaker:Phoa, Frederick Kin Hing (Institute of Statistical Science, Academia Sinica)

  • 2020-09-18
  • Digital Ruling
  • Speaker:  /  Host:


Topic:Efficient Designs for Performance Comparison of Online Experiments  
 
Speaker:Phoa, Frederick Kin Hing (Institute of Statistical Science, Academia Sinica)
 
Date Time:FRI. Sep 18,2020, 10:40 AM - 11:30 AM 
 
Place: 4F-427, Assembly Building I
 

Abstract

Designing an experiment for the comparison of experimental results with different experimental settings has been a classical problem.  As science and technology have advanced nowadays, such problem arises again in the performance comparison between different layouts in the online experiments. Different from the past, the number of adjustable settings in the online experiments is usually very large in this big data era, so that traditional experimental designs easily become infeasible for standard computational capacity. In this talk, we review several traditional approaches that include the multiple comparison (or a/b testing), factorial experiments, and some others, and we point out their limitations for the experiments in the modern days. Next, we introduce a special class of factorial designs called supersaturated designs for efficient experimentation, and point out its problem in estimation.  Then we introduce a new class of cost-efficient design, namely repeated coverage designs (RCDs), that can provide relatively robust estimations on large number of factors with small number of runs. We will provide some comments and hints on the practical use of these RCDs in hardware and software testing, especially in online experiments, at the end of this talk.

Keywords: Online Experiments, Repeated Coverage Designs, Cost Efficiency, Super- saturated Designs