Case Study: E-Commerce Personalization
Working with a Fortune 500 company, built an auto-ml pipeline to assess potential for personalization and publish the best performing approach.
Outcome: Effective Personalization At Reduced Cost
Problem Description
A Fortune 500 company expressed a desire to develop a mechanism to assess the potential for personalization in a systematic way. The client is a large organization with a B2C e-commerce website. The potential to develop personalized customer experiences across their website is large. An A/B test is typically used to compare two or more ways of serving a customer epxerience. When a personalized experience is introduced, it is usually compared using an A/B test to what was there before. This comparison is conducted using busines KPIs. While A/B tests are a great tool for data-driven decision making they have their limitations:
- When an A/B test is conducted web traffic is re-routed and randomly assigned to one of the experienecs being tested. If any of the experiences are loosing revenue, a test might end up affecting the bottom line.
- The more approaches for personalization are compared to each other, the more data needs to be gathered. So testing many options at the same time is also potentially risky and expensive.
- An A/B test only compares two or more things. It does not tell us what it is that's relevant in the customer's decison making
- Within a large organization, customer experiences are tweaked all the time. This means there is typically a much larger need for testing, compared to what can be reasonably accomodated wihout making most of the web traffic part of some randomized test.
The goal of this project was to enable internal clients to evaluate personalization strategies while minimizing the number of live tests that are conducted.
Reliancy Solution
Over the course of three years we worked with an internal team which was tasked to deliver an experimentation platform. Our contributions involved: