Problem Description

Running randomized A/B experiments is the holy grail of causal inference. When conducted properly, they allows us to establish causal relationships. However, a randomized experiment may not always be possible. Furthermore we may want to understand preferences on a customer level, rather than on average. Frequently clients are interested in understanding what drives customer decisions. One of our Fortune 500 clients wondered how we could help them address these types of questions.

Uplift analysis has gained a lot recognition in recent times. It combines causal inference and machine learning to answer counterfactual what-if questions about customers. In particular once a campaign or randomized experiment has been run, we can build models to understand how customers would have responded to different offerings. This is useful because it can be used to drive marketing and business decisions.

Reliancy Solution

We educated the client on causal inference and implemented an applicable pipeline.

• Education On Causal Inference And It's Applicability to Business

  Causal Inference can be intimidating to digest and is non-trivial to apply. Typically part of the process is an upfront understanding of what causes what or a causal graph. This type of graph requires deep domain expertise. However, in the case of randomized experiment there is a simpler way to apply causal inference. We devised a tutorial which was presented to multiple teams.

• Implementation Of Uplift Analysis Pipeline

  Given data from a randomized experiment, we evaluated the "DoWhy" package from Microsoft and the "CausalML" package from Uber. Due to a better support for sparse data we ended up implementing a pipeline using CuasalML.

• Implementation of Actionable Reports Based on Uplift Analysis

  The end result of an uplift anlysis pipeline are treatment effect scores for each customer. We designed resports to illustrate how these treatment effect scores can be readily used to produce more interpretable reports and express the prefrences of customers.

Impact

• Helped Further Understanding Of Causal Inference

  We helped leadership and the team understand where and how causal inference is applicable to their needs. We also helped them understand how it can be effectively used to communicate internal assumptions when doing analysis.

• Provided A Machanism To Ask What If Questions

  We implemented a scalable uplift analysis pipeline that can be used on data obtained from randomized experiments. The pipeline allowed to examine how different segments of customers would have behaved if they were presented with a specific offering.

• Helped Answer Questions From Internal Customers About What Drives Customers

  Part of the value of uplift analysis is that one can form segments by certain variables of interest. For example if a client has the suspicion that zip code is a driver for customer behavior we can assess that. The pipeline we built was used to answer these types of questions that came from internal clients.