Five things every experimentation manager must know about managing a testing program
Experimentation Leads / Managers / Capability Leads have an interesting role to play. On one hand, many of them have been promoted from practitioner and specialist roles and are still getting acquainted with what is required of them.
From our research in the industry, we have seen that a majority of individuals with the role of an Experimentation Manager i.e the one responsible for a team or teams, are still stuck doing practitioner level tasks such as running experiments. Others have elevated to creating strategy and reporting on their teams performance.
There are a lot more that an Experimentation manager should be doing and we have compiled a list of 5 things as a starting point to guide you and make you aware of what you may have not considered or understood fully. We have distilled this from years of working with companies and seeing what works and what doesn’t work. We have looked at what Experimentation Managers have done, the bare minimum and where the bar should be set.
These are
- Set the standards
- Connect the dots
- Understand the “real” performance
- Coach and grow the capabilities
- Engage & Enable Self Service.
1. Define & Set The Standards – Creating Governance & Guardrails
The first port of call for a new experimentation manager or lead is to get the lay of the land and to understand how the experimentation program has progressed in the company so far.
To get a clear picture, these are the areas to dive deeper into
A) Who in the organization is running experiments currently?
B) How are these experiments being run? What is the methodology used in ideation, prioritising, planning and reporting?
C) What processes are being followed? Is it uniform across the board? ie. everyone running experiment plans, prioritises and reports on them in the same format?
D) Where is the data of all the insights and experiments done stored?
E) What is the hygiene of the data being captured? Are all relevant pieces of data captured correctly and not changed later in the process?
F) What is the depth of knowledge and execution regarding experimentation? Is it surface level or are they able to run complex tests independently?
The main aim of asking these questions is to plot out the strengths and weaknesses and areas of improvement.
If you have been brought in to improve standards, knowing which areas the standards are lacking and where it might need only minor tweaks can make all the difference.
If the teams in your organization have operated autonomously and run tests in whatever way they liked, here is your chance to throw down the gauntlet and create a framework for governing the program and the guardrails that keep it in check.
Having a governance framework in place gives your experimentation program an edge over ones that don’t care much for it. It is the system that will keep your team accountable and working within the guidelines that you have set for them with no excuses to deviate from them.
2. Connect The Dots – Align Experimentation With Business Objectives
Experimentation when done correctly has immense power – to give you answers to your questions.
Unfortunately, often, I will come across experimentation programs where all the measure is Conversion Rate Uplifts and Revenue Gained (even though it’s hard to extrapolate an accurate number over a period of time)
Testing becomes spray and pray with any and every tactic thrown in all with the aim of improving “conversions”
However, as a lead, understanding the objectives and goals of the senior stakeholders and connecting it to experimentation and research activities will allow you to paint a clearer picture when it comes to presenting it back to stakeholders.
Too often, experimentation teams get stuck in a perpetual struggle of having to prove value of their testing programs and run spray-and-pray tests in the hope to improve conversion rates.
This results in the stakeholders questioning the impact of testing even when they are positive results on show.
The approach of working closely with stakeholders, understanding their goals, KPIs and objectives is a skill in itself and time consuming but must be done.
Once you have all the objectives, you need to create a map of all these objectives and prioritise the research, ideas and experiments that tie to and support that objective.
When it comes to reporting to that stakeholder you can show the various supporting activities that helps them with that specific objective.
3. Understand the “real” performance
When asked about performance, it’s easy for an experimentation manager to look at vanity metrics as the indicator of a teams performance – number of tests run, number of winning tests and revenue gained.
The truth about experimentation is that you can’t really control the outcome of the test (try as you might think otherwise) but you can control whether the test is reliable enough to learn from.
This means checking performance against the standards set and how well the processes were adhered to. This will ensure that the test was planned reliably against that criteria and if there are any red flags, it should get picked up.
The second part of this is in the reporting aspect and it’s here where experimentation managers have to keep a closer eye on.
The phenomenon of HARK-ing (hypothesizing after the results are known) can happen easily and can pass by undetected. HARK-ing can take the shape of cherry picking what metrics to report on, changing goals or hypotheses to align with the findings of the experiment, changing the key definitions within the experiment plans to reshape the narrative of the experiment outcome.
All of this usually happens when there is a pressure to perform and drive higher conversion rates and revenue uplifts.
An experimentation manager who is keen on improving standards in an organization will focus on monitoring every aspect of the experiment process from planning to reporting.
This is where the concept of an Experiment Health Score can help the experimentation manager standardise the monitoring of the performance.
4. Coach & Grow The Capabilities
As an experimentation lead, you will need to train and onboard people to run their own experiments. It’s easy to think that giving them a training session or a workshop, access to the toolset and sending them on their merry way will be sufficient. It’s not.
[[Insert image about change]]
A person or a team getting onboarded is only the beginning. This is where experimentation managers can drop the ball. If there is no system to monitor, checkin and address any issues early on, chances are this onboarding will fail in a few weeks. Regardless of the enthusiasm shown by people in the early days, the benefit is negated when theres no mechanism to spot issues early on and coach the individuals to improve.
As the number of teams involved in experimentation starts to scale up, bad practices, if left unchecked, will also scale up leading to chaos. Having a clear oversight of every individual and team is time consuming but the lack of it will create more issues.
If the experimentation manager aims to foster a coaching mindset, they must consistently review the work done by the teams at regular intervals and give feedback on areas of improvement.
5. Engage & Enable Self Service
Experimentation Teams struggle with getting buy in and engagement from stakeholders and the wider business. The challenge remains because they have done the same thing over and over again.
“We share spreadsheets of all our tests”
“We present all the experiments in our hands on meetings”
These activities may seem like they move the needle but they don’t. They are check box exercises that give the illusion of progress but don’t really drive meaningful engagement.
The Experimentation Manager should not delegate the tasks of engaging the wider organization to the practitioners. Instead, they should be the ones building relationships and facilitating the easy access of insights and results.
Encouraging and enabling stakeholders to self serve has a positive impact because it removes the bottleneck of CROs being gatekeepers of the data. It enables stakeholders to utilize the data and insights in decision making.
Having insights at the ready and being able to self serve will help CRO teams as they are no longer constrained having to find data for them everytime.
Going beyond report creations and emails just because it was done that way in the past is where an Experimentation manager can make a big difference. If they can challenge the status quo (because the status quo) isn’t working can help drive change
There is another factor that plays a bit part in whether stakeholders will engage or not and that is whether the data is in a “language” that they can understand.
Practitioners tend to create reports in a way they understand it, full of jargon, stats and numbers which only get a glancing look by others in the business. Time spent creating beautiful powerpoint decks lie gathering digital dust. This is a waste of everyones time and resources.
If the Experimentation Manager implements the concept of structured insights that can summarise the learnings and outcomes and make it relevant to the person reading it, they will have a greater chance of getting them to engage back.
