Experimentation can open up a world of opportunities, but there's a practical limit to how many experiments you can run concurrently, primarily dictated by your user base or traffic. Running a multitude of experiments without adequate users can lead to inconclusive or misleading results. Hence, the goal isn't to execute as many tests as possible, but to judiciously choose the right experiments that maximize learning and potential uplift.
Among the numerous prioritization frameworks available, the ICE (Impact, Confidence, and Ease) framework stands out for its simplicity and effectiveness.
1. Impact: How significant is the expected change on the key metrics? 2. Confidence: How sure are you that this change will lead to a positive outcome? 3. Ease: How much effort, time, and resources are required to implement the experiment?
Each element is typically scored on a scale from 1 to 10. The average of these scores gives an overall priority score, guiding which experiments to run first.
The effectiveness of any framework lies in its application. Adapting and tweaking the framework to fit your context can lead to more informed decisions. Here are some points that we use internally to make the most out of it:
<aside> 💥 Adjusting Impact The position of a change on a webpage can be pivotal. If the alteration is above the fold—immediately visible without scrolling—we add a point to the Impact score. This modification reflects the potentially heightened visibility and influence of such changes on user behavior.
</aside>
<aside> 📊 Boosting Confidence with Benchmarks Learning from the broader market can be invaluable. If a competitor or another industry player has implemented a similar change, it can serve as a point of reference. Recognizing this, we add a point to the Confidence score, drawing assurance from the existence of industry precedents.
</aside>
<aside> 🤔 Rethinking Ease When considering how easy it is to run an experiment, we don't just look at the full-blown implementation. Often, there's a quicker, "hacky" way to test a hypothesis without diving into a full commitment. Think of it as taking a shortcut to gauge potential changes. If it turns out the full implementation might be expensive, we can use our test results to decide if it's worth the effort. This lean approach to experimentation helps in making informed decisions without overcommitting resources from the get-go.
</aside>
All organizations operate differently; thus, it's unrealistic to assume that a single prioritization model will be universally effective. It's essential to recognize that no decision-making process is completely objective or without its flaws, but that's acceptable.