CONTEXT-AWARE CHOICE OF METRICS IN A/B TESTING

Abstract

The study is aimed at improving the methodology of selecting metrics for A/B testing by introducing an extended set of metric attributes that can serve as the main search criteria for determining the appropriate metrics for a particular experimental context. Along with such parameters as sensitivity, latency and noise, structural characteristics (e.g., data type and metric aggregation scheme, associated business process) are proposed to be included in the evaluation system. By summarising the published results of numerous studies, the values of key metric parameters such as sensitivity, noise and latency are ordered, a hierarchical model of interdependencies between metrics of different levels is built, and the procedure for forming a set of specific metrics for A/B testing using a metric tree for the correct identification of surrogates is determined.