Using Multi-Armed Bandits During the Holidays

This is why so many of our customers institute a Holiday Freeze (a topic on which we also have thoughts). Black Friday and Cyber Monday are table stakes and the risk of running a failing variation is not a viable option for many companies. Not to mention this Holiday Season is but 26 days. But, what if you had the opportunity to optimize rather than experiment? 

Optimizely’s optimization feature, Multi-Armed Bandits, is designed to allow users to capitalize on the best performing variation(s), in order to recognize more conversions. This optimization automatically funnels more visitors to the best-performing variation on an hourly basis. I will caveat this by saying – Multi-Armed Bandits is not an experiment, so you will not have statistically significant results to share with your team, however, you may get a pat on the back for maximizing revenue or conversions. 

Imagine the following scenario: Your teams are gearing up for holiday sales and promotions, anticipating record sales numbers, betting on copy and images to get just a little more sold than last year. Disagreements arise around the marketing plan: what are we actually going to display on the live site for Black Friday? What if machine learning could make that decision for you?

A Multi-Armed Bandit takes the variations presented (including the original) and treats them all independently, with no concept of a baseline to compare against. The bandit will observe the conversion rates on each “arm” and will start to allocate more traffic to the one performing the best (as defined by the primary metric). 

Optimizely’s algorithm continues to check the conversion rates hourly, so with soaring incoming traffic during the holiday sales, you can capitalize on those conversions. The lower performing variations will continue to get a small amount of traffic so that conversion rates can continue to be observed in case of a shift in user preference.

A few examples of how to apply this:

• Before Black Friday and Cyber Monday approach, you are encouraging users to get reminders and sign up to watch items. Using Optimizely’s Multi-Armed Bandit, you pose these options in a few different ways:
  •  A pop up letting users know you noticed them checking out an item, and giving the option to watch the item.
  • The next option presents this option in a different way, maybe highlighting the item with an icon to watch it. The pop up is working better so more people will see this version, allowing you to get many more “watchers” than you would have if you had split traffic equally. 
• On Black Friday, you have decided that you will feature some of your best-selling items in different categories (e.g.: TVs/accessories/Computers). But how do you influence the most clicks? By using Multi-Armed Bandit, you can make sure the imagery and copy getting the most clicks is the one exposed to the most users. 
• Cyber Monday is approaching. You’re an education company with online courses available for hobbies and continued education. You’re encouraging your users to sign up for classes by offering them at discounted prices. You have debated internally about presenting a countdown for time to the end of the sale. Another idea was to present the courses in bold tiles. You can do BOTH and a combination of the two as another option. Multi-Armed Bandit will help drive those registrations/purchases.

At this point, you’re getting excited, thinking about all the ways that you can use the Multi-Armed Bandit to gain more revenue or increase your conversions. I get it- I’d be excited too. Some things you should know before you dive in…

Multi-Armed Bandit is available to folks on Optimizely Web and/or Optimizely Full Stack Enterprise plans. This is the highest tier Optimizely offers. If you’re not sure if you have this, your account team can help you. 

Optimizely is an experimentation platform first and foremost, but it is important to note that this is not a tool to experiment. The results for a Multi-Armed Bandit will focus on the performance of your primary metric, with no statistical rigor involved. I do not recommend using this to learn about how to format your navigation, a product listing page, or a lead form. Things that live permanently on your site should be tested with statistical significance telling you that the response from the users you are seeing are not due to chance. 

The holidays can be a stressful time for us all. We are worried about travel logistics to see our families, what gifts we will get our loved ones, and what food we are going to make or eat. Work stress can pile onto this. Multi-Armed Bandit can take one thing off your plate, eliminating the interoffice debates, and finding the best variation to convert your users for you!

If you have questions about how to use Optimizely’s Multi-Armed Bandit or if your use case is a good one for this type of optimization, I’d love to hear from you. You can always ask your account team, or shoot me a line (caitlin.hutcherson@optimizely.com). 

Happy Holidays!