The Thomke Talks Episode 4: The ROI of experimentation

People often ask about the ROI of experimentation.

What I tell them is that experimentation is kind of the the enabling technology.

You know, it's like asking, what is the ROI on breathing? You know, it's it's one of the things that you kind of have to do if you want to manage. In fact, I would argue that innovation or being innovative is actually the lifeblood of of of organic growth. The ROI can be measured in a lot of different ways, but if you're just waiting and waiting until you can see some attractive number, it's gonna be too late because the ROI will also depend very much on how you implement these experimentation platforms within your own organizations.

I tell people, get on it. I mean, you better get started and try to do it at scale, not just, in a little corner over here, try to get a lot of people sort of involved. In fact, you know, in some of the organizations, true experimentation organizations, whose culture, whose management, processes have fully embraced this, they have broad involvement. You know, some organizations were fifty, sixty, seventy percent of all employees involved in designing and running experiments.

That's the kind of level you want to get to. Then experimentation becomes like, breathing, like running the numbers, you know, nobody's gonna ask for the RA because the RA is so huge. You know, you can ask the opposite question, why the cost of not doing it. And I think the cost of not doing it is actually even more expensive because you're going to be at a competitive disadvantage.

Engineers at Yahoo sort of clearly said, and it's been on the record, saying the reason why Google was beating Yahoo is because they simply are random. They all ex I would experiment with them. And, so it becomes that competitive issue, not just the ROI issue.