Investing in Optimization – The First Steps Towards Building a Team

Her latest challenge? Building an optimization program from the ground up, and enabling her organization to unlock the power of rapid testing while working towards an aggressive revenue goal. In this series of three posts, we’ll cover how she approached team structure and process, how the team set goals, and examples of experiments her team has run.

Move, Inc. is a provider of online real estate services for consumers and real estate professionals to help navigate all stages of the home-buying cycle. The company is heavily invested in understanding its target audience, with teams devoted to both analytics (measurement of visitor behavior), and research (focus groups and usability studies).

Suma arrived at Move on the cusp of transition: the Analytics team had decided to renew their focus on optimization. They saw optimization as an ideal method for taking action on what they were learning from qualitative and quantitative study of their visitors. They chose a new optimization platform (Optimizely) and began to staff a new team focused on rapid experimentation and optimization.

In 2014, Suma and her team ramped up ideation, prioritization, execution and analysis of Realtor.com’s tests. In this post, we’ll cover how the team is structured, the importance of their cross-functional stakeholders, and the agile processes they adopted to ensure a smoothly running optimization program.

Structuring an Optimization Team: Aligning Strategic Resources

At Move, the Optimization team is housed within the Analytics department. For the team to be successful, they needed cross-functional support from Research, Design, Product, QA and Engineering. The core Optimization team included Suma, Roger, and Hima to manage the test prioritization, building and running tests, and post-experiment analysis. Two key technical assets that they focused on assigning immediately were QA and developer resources.

Sometimes, optimization programs can face roadblocks within engineering and QA organizations. This is because fast-paced optimization requires rapid changes to production code, which can be viewed as a threat to the stability of the site. Suma knew that in order to build a successful program, they would need to ensure their engineering peers that optimization wouldn’t pose any risk to the production environment.

Suma knew that assigned QA resources were an important element of the optimization process, as well as development resources to help with more complex experiments. To ensure that her team had strong cross-functional communication with Engineering, she had managers approve ad-hoc support for the Optimization.

“We wanted to make sure the development team members were aware that we wouldn’t change anything without their knowledge, and that our experiments would undergo the same rigorous QA process as the rest of our code.”

Communicate, Communicate, Communicate

Beyond securing resources, Suma wanted to make sure anyone on the technical teams at Move could have access to the optimization team’s priorities and day-to-day activities if they had questions or wanted more information. To foster this transparency, the optimization team created an internal wiki and e-mail listserv to keep all interested individuals and teams in the know.

Move’s internal wiki provides a location for all employees to see current optimization activity and a test idea backlog.

Suma focused next on how her team would interface with design and product. The primary method for keeping this group tightly knit and communicating frequently was regular stand-up meetings. The core team would meet to review tests in progress and results of recently concluded tests. They would also discuss prioritization and building of new experiments. Face-to-face communication ensured that all stakeholders had an opportunity to voice issues and ensure that no experiments were conflicting with each other.

Buy-in from design and product would also become an important step in evaluating and prioritizing potential experiments. Why? The optimization team focused exclusively on test ideas that would actually be implemented if successful. If an experiment would, for any reason, not be approved by design or implemented by product, the experiment idea was quickly deprioritized.

“The value of optimization was there, but we wanted to understand how to speed up testing, how to build a high-caliber team. The secondary pillars of how to communicate, how we socialized our optimization goals, were all part of our strategy to make sure we reached those goals.”

As the optimization team became comfortable with their communication and process, speed and cadence became a priority. In order to take advantage of the benefits of optimization, Suma wanted to ensure that the team could rapidly test many ideas, quickly make decisions, and incorporate customer data into their process. By laying a strong foundation, the team was able to reach a pace of launching 1 new experiment every 2 weeks.

Suma credits the Move team’s success with rapid testing to two factors. First, the optimization tool (Optimizely). A key criteria in choosing the team’s tool was finding one that would enable rapid testing and empower the end-users. This would become very important to ensuring the team would meet their optimization goals, but also showcased the value of switching to a new tool.

The second factor was the team’s focus, which allowed them to ask more questions about their customer behavior and find answers faster while moving the product as a whole in the right direction. As Suma puts it, the risk of not moving quickly is that ideas will be deliberated, but never executed. Their team took the opposite approach and was able to reach tangible, quantifiable goals for their business.

In the next installment of the Move story next week, we’ll go into how the team set an aggressive optimization goal and made their success story visible by celebrating their wins.