Experimentation under pressure: How to survive (and thrive) in a world of budget cuts and rapid results

Remember when A/B testing was as simple as changing a button color and watching the conversions roll in?  

Yeah, those days are gone just like flip phones and dial-up internet. In 2024, most testing programs are trapped in a wild reality show called "Extreme experimentation: Budget cut edition."   

The focus is now on demonstrating tangible value and ROI, fast. This change brings with it several key challenges:  

• Brutal budget cuts  
• Misaligned priorities 
• Impossibly tight deadlines  

However, the power of experimentation lies not in the program itself, but in how human decision-makers interpret and act on the data it provides.

Let’s embrace that test and learn mindset and look at some ways to get your program back on track through data driven decision making. 

The budget crunch: Surviving the fiscal hunger games

To prove value, you need resources. And for more resources, you first need to prove value. Every quarter, you and your experimentation program are expected to solve actual business problems in real time.

Here's how you can make it happen:

1. Build those relationships

   Not just with your boss, but with wider stakeholders including the CFO. Have full context on how your business is performing (redundancies, cuts, market pressures, quarterly forecasts, and competitor moves). The more intel you gather, the better you can position your experimentation efforts. 

   For example, know the business facing pressure to reduce churn? Prioritize retention experiments.

   

   Image source: Optimizely

   Know thy stakeholder 

   Start by tracking who's who, their influence, and their stance on experimentation. Ensure you communicate the value of experimentation in a variety of formats tailored to your target audience (Example: Teams updates, emails, meeting updates and more) 

   Example: 

   • For the C-suite: This program increased revenue by X% last quarter. 
   • For the dev team: We reduced bug reports by Y% through systematic testing. 
   • For HR: Our latest test improved employee satisfaction scores by Z points. 

2. Embrace value based budgeting

   Every dollar needs to justify its existence. It's not just about cutting costs; it's about maximizing impact. 

   Before greenlighting any experiment, estimate its potential value in numbers through projected revenue increase, cost savings, or conversion rate improvement.

   Set minimum thresholds for experiments, and allocate budget dynamically based on proven value.

   

   Image source: Sketch bubble 

3. Prioritize, prioritize, prioritize

   Prioritize like your program's success depends on it (because it does). Your program should be a mix of quick-hit tests for immediate gains and longer-term experiments for sustained success. 

   Start with simple, high-velocity tests to build momentum and prove value. Then, run complex experiments as you build credibility to deliver high impact. 

   You can even get creative. Repurpose old tests, squeeze more insights from existing data, and look at every resource as a potential testing tool. 

The time squeeze: Maximizing testing efficiency in a rush 

When "I need it yesterday" becomes your boss's mantra, it's time to optimize every experience. Here's how to squeeze more productivity out of your day: 

1. Streamline your processes

   Audit your experimentation workflow and eliminate unnecessary steps. Replace status meetings with shared dashboards, use templates for common experiment designs, and automate result notifications. Your goal is a lean, efficient workflow that cuts through the clutter. 

2. Embrace rapid testing

   Adopt approaches like painted door tests to quickly validate ideas before full-scale development. Add a button for a non-existent feature to your site - if people click on it, you might be onto something. This method can save weeks of development time on unviable ideas. 

   Remember... As fast as possible, as slow as necessary. 

3. Statistical significance

   While velocity is important, don't forget statistical significance. Set clear minimum sample sizes for different test types, use sequential testing methods to potentially conclude tests earlier, and weigh the cost of a wrong decision against the benefit of moving quickly.

   

   Image source: Optimizely 

   By streamlining processes and embracing rapid testing while maintaining statistical validity, you'll generate insights at lightning speed without sacrificing quality. 

When prioritization matrices meet reality

Sometimes, your carefully crafted roadmap gets hijacked by the HIPPO (Highest Paid Person's Opinion) in the room. 

First, recognize that HIPPOs aren't inherently evil. They're often driven by: 

• Pressure from stakeholders 
• Market trends they've observed 
• Gut feelings based on experience 

Your job is to translate these executive instincts into testable hypotheses. To do this effectively, incorporate "HIPPO-friendly" factors into your prioritization framework. Consider elements like strategic alignment, executive interest, and potential PR value. By framing your experiments in these terms, you're not fighting against the HIPPO's influence - you're keeping your experiments aligned with business objectives while satisfying executive priorities.

Here's how to further stay on track: 

1. Master the art of diplomatic pushback

   When hit with a new idea request, don't just say no. Instead: 

   • Listen to test, not reject: Ensure you’ve heard and fully understood the request 
   • Acknowledge the idea's potential: I appreciate you sharing this idea. How do you see it fitting into our current strategy? 
   • Align with objectives: How about we modify the idea to better fit our Q3 targets? 
   • Propose a pilot: Suggest a small-scale test to validate the idea before full commitment. 
   • Present data-driven alternatives: Have you considered this approach? Our data suggests it could achieve similar goals with less resource investment. 

   It's not about winning. It's about finding the best path forward. Sometimes, that means meeting in the middle. 

2. Make prioritization matrix work for you

   If you can't beat 'em, join 'em. Incorporate factors like "executive interest" or "strategic alignment" into your prioritization framework. This way, you're speaking the language of leadership while maintaining some order in your program.  

   Your matrix might include: 

   • Potential impact (quantitative) 
   • Resource requirements 
   • Strategic alignment 
   • Executive interest 
   • Risk level 

   By giving weight to these factors, you create a prioritization system that balances business needs with data-driven decision-making. 

   Overall, prioritization in the real world is often more art than science. For success, balance data-driven decisions with strategic flexibility. 

The shipping vs. value dilemma

Product teams are expected to ship the right ideas fast. Product managers are often caught between the rock of "ship it now!" and the hard place of "but is it valuable?" 

Let's dive into how to balance those short-term wins with a long-term experimentation strategy that actually moves the needle.

Start by implementing a "value-first" approach to feature development. Before green-lighting any new feature, ensure there's a clear hypothesis about its potential impact and a plan to measure it. Define a specific outcome for your target user, outline measurable success metrics, and plan your measurement strategy. Remember, if you can't measure it, you can't improve it.

Next, master the art of the minimum viable experiment. Can you test the core assumption behind a feature without fully building it? Identify the riskiest assumption, design the simplest possible experiment to test it, and set a clear threshold for success. This could involve landing pages, prototypes, or even a well-placed button that goes nowhere (hello again, painted door tests!).  

In the eternal battle between shipping and value, the real winner is the team that figures out how to do both. By focusing on rapid, value-driven experiments, you're not just shipping – you're shipping smart by balancing the need for speed with the pursuit of genuine impact. 

Managing the never-ending backlog

Use the below strategies to turn that mountain of ideas into well-oiled features and products: 

1. Backlog curation

   Think of your backlog as a garden. Regular pruning isn't just good – it's essential. Schedule regular sessions and ruthlessly evaluate each idea. Does it still align with current goals? Has its potential impact changed? If it doesn't spark potential value, thank it for its service and let it go. 

2. The "one in, one out" policy

   For every new experiment idea that gets added, another must be removed or deprioritized. This not only keeps your backlog manageable but also forces you to continually reassess priorities. 

3. Set clear expiration dates

   If an idea hasn't made it to the top of the priority list after a certain period (say, 3-6 months), it's time to say goodbye. This doesn't mean the idea is bad – it just means it's not worth building right now. 

   Remember, if everything is a priority, nothing is – and in the world of feature experimentation, focus is your secret ingredient. By implementing these strategies, you'll stay focused on high-impact, timely initiatives. 

Strategies for success

Now that we've identified the challenges, let's arm you with some practical strategies to not just survive, but thrive in this new landscape.  

1. Become a master storyteller

   Data won't speak for itself as most won't know how to interpret it. Be the data’s spoke person and turn those dry metrics into actual business impact stories that'll make even accounting teams sit up and listen. Craft narratives that resonate with different stakeholders. You're not just sharing results; you're selling a vision.  

2. Breakdown those silos

   Identify allies across the organization who understand the value of experimentation. These are your secret agents in your company. Meet regularly and share wins so when you need to push a bold new initiative, you'll have a ready-made squad to back you up. 

3. Embrace agile experimentation

   Break those mammoth experiments into smaller, iterative tests. Quick wins keep the momentum going and the stakeholders smiling. Make sure your tools are flexible enough to keep up with your agile mindset. 

4. Invest in knowledge-sharing

   Create a centralized repository of experiment results, learnings, and a/b testing use cases. It's like building your ownexperimentation Wikipedia. This not only improves efficiency but also demonstrates the cumulative value of your program over time.  

   Optimizely experiment collaboration can help you build collaboration and share knowledge across your organization as not everyone feels comfortable getting involved in testing straightaway. Through standardized workflows and shared plans, you can keep everyone engaged and capture great ideas.

   

   Image source: Optimizely

5. Approach experiments with design thinking

   Empathize with users, define problems clearly, ideate, prototype rapidly, and test relentlessly. Learn more about product ideation techniques. 

Wrapping up: Moving forward in tough times

Let's recap the key strategies we've covered for handling challenges in experimentation: 

• Budget efficiency: Explored ways to maximize impact when resources are limited. 
• Accelerated testing: Learned techniques to increase velocity without compromising quality. 
• Strategic balance: Discussed how to align company goals with sound testing practices. 
• Feature management: Saw tips to balance new feature development with demonstrating value. 
• Workload optimization: Covered methods for managing extensive backlogs and heavy workloads. 

Remember, while the current climate presents challenges, it also offers opportunities for innovation. With strategic thinking and the right approach, you can not only navigate these obstacles but also uncover new insights and drive value for your organization. 

No matter the strategy, you still need the right tools to help you. With Optimizely's experimentation capabilities, you cancollaborate with everyone and validate ideas in one place.

It allows your team to deliver results without code and test across any channel so you can target with precision.

Sound good? See Optimizely Web Experimentation and Feature Experimentation in action.