An A/B Test Validates New Business for uSell.com
uSell.com is an online marketplace for customers to sell their smartphones and other used electronics. To gauge customer interest in selling new products to validate business expansion plan, the team at uSell ran an A/B test measuring conversions on electronics with addition of new products to sell. This is the story of that test, the
uSell.com is an online marketplace for customers to sell their smartphones and other used electronics. To gauge customer interest in selling new products to validate business expansion plan, the team at uSell ran an A/B test measuring conversions on electronics with addition of new products to sell. This is the story of that test, the results and the key takeaways moving forward.
Challenge
Looking to expand the business beyond cell phones and electronics, uSell surveyed customers asking what other types of products they would be interested in selling on the site. Through qualitative surveys and measuring click-through rates in marketing emails, the team observed that customers were interested in selling items such as textbooks, clothing, and video games on uSell.com, but they wanted data to prove it.
“We were still nervous about disrupting conversion on our core vertical — electronics,” said Chief Product Officer Christian Croft. “We needed to prove that we could provide more opportunities to the customer without distracting them too much from our primary offering.”
Hypothesis
The uSell product team hypothesized that they could add additional verticals to the site without affecting their core conversion rate. Before taking a risk, however, the team ran an A/B test to validate their hypothesis.
The Test
To test this hypothesis, the product team built simple landing pages for the new product categories. Then they designed an alternate version of the homepage that featured the 4 new product categories. Visitors in the original A version of the test came to the standard site — with only the opportunity to sell electronics. Visitors bucketed into the B version saw the additional categories — textbooks, gift cards, women’s clothes, games, and children’s clothes — as icons on the homepage, and as additional links in the header menu. Within Optimizely, they used a substring match in targeting to apply the navigation change to all pages in the variation experience.
A
The original version of the site allows customers to sell eight different types of electronics by clicking on the icons or the header menu links.
B
The variation experience features the new sellable items as icons and in the header links menu.
They focused on customers placing orders to sell their electronics as the barometer of success in the test. As long as the variation showed no significant negative impact to the core conversion, they would make the expanded products the default. Orders from other verticals would add incremental revenue worth pursuing for the business, the team decided. “In Optimizely-speak, our Order Confirmation goal only has to show an even conversion rate between Original and Variation 1 for us to be high fiving,” Croft said.
They established a threshold of success for the test: in order to count as a win, the variation could not detract more than 5% from conversions on electronics.
Results
After running the test twice, they found that adding additional verticals to the site had somewhere between a break even and -3% impact on conversions on the electronic business. They determined that the potential for growth and the PR impact outweighed the slight decrease in conversions.
“Instead of being focused only on selling electronics, we can commit to offering consumers the opportunity to sell a wider variety of products in our marketplace.” Croft said.
With the validation that allowing customers to sell a broader array of items was a boon for business, uSell made the expanded verticals and navigation a permanent aspect of their website.
Takeaway
Test to validate hypotheses. uSell had an idea for expanding into an entirely new business model, but needed to back it up with data. Testing helped them eliminate project risk and prove the validity of their idea through testing. In a couple days of running a test, uSell was able to validate a business idea that would bring incremental revenue to the company.
Apply insights across channels. uSell airs TV commercials about its service. To date, those had focused solely on electronics but armed with the data from this A/B test, they can broaden the message on commercials on selling a wider variety of things, Croft said.
