Focus

Hiring and Training an Experimentation Team: Roles, Skills and an Upskilling Plan

Alexandre Suon · 2026-10-01

Testing tools are easy to buy. The people who can run trustworthy experiments, and a business that knows how to use them, are not. This deep dive covers how to build an experimentation team: the roles you need, the skills to hire for, when to hire, outsource or use a fractional team, how to interview, and a training plan that spreads the skills across the business.

Executive summary

1. People and skills, not tools, are what limit most programmes. 47% of marketing decision-makers cite a lack of resources as a main A/B testing challenge (Ascend2, 2025), 52% of respondents to Speero's maturity benchmark admit they do not QA tests before launch, and when Booking.com began measuring experiment quality, 80% of its experiments had no power calculation.
2. Seven roles cover the work, but you rarely start with seven people. An experimentation lead, a programme manager, an analyst, a developer, a UX researcher, a designer and a product owner. Early on one person holds two or three of these roles, and champions in other teams extend the reach without being hires.
3. Size the team to your maturity and traffic, not to your ambition. Microsoft's Experimentation Evolution Model moves from a standalone central data science team running a handful of experiments to product teams running thousands on their own. In our experience a programme running 10 to 99 tests a year needs a core of three to five people.
4. Hire what is core and constant; rent what is specialist or occasional. The lead, product ownership and customer knowledge should sit in-house. Server-side builds, audits and research sprints can be bought by the project, and a fractional team bridges the gap while you hire. The US Bureau of Labor Statistics projects 35% growth in data scientist jobs from 2025 to 2035, so analysts will stay hard to find.
5. Interview on a real result, not on tool trivia. A 45-minute exercise in which the candidate reads a flawed test result (with a sample ratio mismatch hidden in it) tells you more than any certificate.
6. Train every audience for the decision it makes. Executives need one hour, product owners half a day and two coached tests, analysts two to three days of statistics. Tie training to a "licence to test", review a sample of experiments, and build career paths so the people you train do not leave.

Section 1 · Why it matters

The bottleneck in most programmes is people and skills, not the testing tool

An experimentation team is the group of people who plan, build, run, analyse and decide on controlled experiments such as A/B tests. It usually combines an experimentation lead, analysts, developers, designers, researchers and product owners, and it may be a dedicated central team, a centre of excellence supporting product teams, or a mix of in-house staff and external partners.

Every experimentation programme has a tool. Few have enough people who can use it well. The licence is a purchase order; a trustworthy programme is a set of skills: framing a hypothesis from evidence, sizing a test, building a variation that does not break the page, spotting a broken split, reading an interval honestly and persuading a product owner to act on a loss.

In Ascend2's 2025 survey of 402 marketing decision-makers, 47% named a lack of resources (time, tools, budget) as a main challenge with A/B testing, and 38% said tests take too long to set up, run and analyse. In Speero's 2025 maturity benchmark, in which 154 respondents assessed their own programmes during 2024, 52% admitted they do not QA tests before launch and only 15% said they have a well-resourced set of research methods (agency survey data, sponsored by testing vendors). In CXL's 2020 survey of 399 optimisers, roughly 56% had three years' experience or less.

When Booking.com started to measure the quality of its experiments, its data science team found that 80% of experiments did not have a power calculation. Booking's answer was not to centralise control. It was education, embedded experts and peer review, which Section 7 covers.

What this shows. The figures come from different populations and years, so do not add or compare them directly. Read together, they say the same thing: most programmes are short of people with method skills (power, QA, research) and short of capacity. Buying a better tool fixes neither. Hiring well, and training the people you already have, fixes both.

If you are still making the case for a programme, start with the business case for experimentation. If the programme exists but stalls, the problem is usually the team, and this article is about building it.

Section 2 · Roles

Seven roles cover the work, but one person often holds two of them early on

An experimentation team needs to do four things well: find problems worth testing, build and measure tests that can be trusted, decide what to ship, and run the machine that keeps this going. Exhibit 2 maps the roles that do this work. They are roles, not headcount: in a small programme the lead is also the programme manager, and the analyst may also set up the tool.

What this shows. The lead and the programme manager sit across the whole lifecycle; the other roles map to a stage. The dashed box matters: champions in other teams are not hires, but a programme that relies only on its own staff will never reach the whole business. How champions are found and organised is covered in where to start: champions, sponsor and rituals.

The analyst deserves a special word, because it is the role most often missing. Netflix describes its experimentation data scientists as working across the whole lifecycle: "data exploration and ideation; designing and executing tests; analyzing results to help inform decisions on tests; synthesizing learnings from numerous tests". The role needs "a broad technical skill set, a self-starter attitude, and a deep curiosity about the domain space". You do not need Netflix's scale to need that profile.

Section 3 · Skills

No one has every skill, so hire for complementary profiles rather than unicorns

Job ads for a "CRO specialist" often ask for statistics, JavaScript, UX research, design and stakeholder management in one person. That person is rare, expensive and hard to keep. A better approach is to define the skills the team needs, then hire people whose strengths complement each other.

What this shows. Every row has at most two expert cells: that is the specialism you hire for. The "aware" cells matter as much, because they are what lets people work together: a developer who understands why sample ratio matters, a designer who knows a test must isolate one change. Use the matrix to spot the gap in your current team before you write a job ad.

Three skills run across every role and are the hardest to train later:

• Comfort with being wrong. Most tests do not win: Optimizely reports that only 12% of experiments produce a statistically significant improvement on the primary metric (vendor data, 127,000 experiments). More on this in how to build a culture of experimentation. People who need their idea to win will bend the analysis.
• Analytical thinking. The World Economic Forum's Future of Jobs Report 2025 finds analytical thinking is the top core skill for employers, with seven out of ten companies considering it essential.
• Clear writing. A test that nobody understands does not change a decision. The one-page result summary is a core deliverable, not an afterthought.

Our view. Hire the lead for judgement and communication, the analyst for statistics and scepticism, and the developer for robustness. Tool knowledge is the easiest skill to teach: any competent developer learns a new testing tool's editor or SDK in days. The reverse is not true.

Section 4 · Team size

Team size should follow maturity and traffic, not ambition

The most cited model of how experimentation grows comes from Microsoft. Fabijan, Dmitriev, Olsson and Bosch's Experimentation Evolution Model (ICSE 2017) describes four phases, Crawl, Walk, Run and Fly, each with a different way of organising data science. The study grouped products by how many experiments their teams run a year (1 to 9, 10 to 99, 100 to 999, and 1,000 or more), and places the Run phase above 100 experiments a year and Fly at thousands. Kohavi, Tang and Xu use similar maturity phases in Trustworthy Online Controlled Experiments. We add the core team we typically see at each phase.

What this shows. The top two rows draw on Microsoft's research (the volume bands are the study's four groupings, which we align to the phases); the bottom two are our view from programmes we have staffed. The important pattern is the shift in the central team's job. At Crawl it runs every test. At Run and Fly it trains, reviews and maintains the platform, while product teams run their own tests. In Microsoft's model, local data scientists at the Run phase are trained as operational data scientists by the central team. Training is part of the job description of a mature central team.

Two constraints keep team size honest. Traffic caps how many tests a site can power, whatever the headcount (the sample size and duration guide shows how to calculate the ceiling). Decision capacity caps their value: adding developers to a programme whose product owners never act on results adds cost, not value.

Which operating model to use at each phase (centralised, centre of excellence, hub-and-spoke or federated) and who owns what is a separate decision, covered in who owns experimentation: choosing an operating model.

Section 5 · Build, buy or borrow

Hire what is core and constant; rent what is specialist or occasional

Every capability your programme needs can be hired, bought by the project from an agency or freelancer, or borrowed from a fractional team: senior specialists who work for you part-time under a retainer. The right answer differs by capability, not by programme.

Disclosure: Henkan & Partners provides fractional experimentation teams and agency services, so we have an interest in this choice. The decision tree below is the one we use with clients, including when the answer is to hire.

What this shows. Two questions do most of the work: is the need constant, and is it knowledge you must own? The lead and product ownership almost always end up in-house, because they hold the customer knowledge and the decisions. Specialist build work, such as server-side set-up or a complex single-page application, is often better bought. Whatever you buy, keep the backlog, the repository and the metric definitions in your own hands, or you will lose your memory when the contract ends.

The labour market favours renting analysts and hiring early

The roles that feed an experimentation team are in demand. The US Bureau of Labor Statistics (BLS) projects employment of data scientists to grow 35% from 2025 to 2035, against 3% for all occupations. Software developers, QA analysts and testers are projected to grow 10%, market research analysts 7%, and web developers and digital designers 5%. These are official occupation categories, not CRO job titles, and the figures are for the United States only.

What this shows. In the United States in May 2025, the median data scientist earned $120,230 a year and the median software developer $135,980, according to the BLS. Pay in Europe, the Middle East and Africa or India is different, so use local data before you budget. The direction is the useful part: analysts will stay scarce, so it pays to hire one good analyst early, train others internally, and borrow senior statistical review when you need it.

Section 6 · Hiring

Test candidates on a real result, not on tool trivia

Certificates and tool logos on a CV tell you little. What you need to know is whether the candidate can think like an experimenter: sceptical of good news, clear about what a result does and does not show, and able to explain it to someone who is not a statistician. A structured interview with a practical exercise is the most reliable way we know to find out.

Interview questions by role

A 45-minute practical exercise, with a worked example

Give every analyst, lead or programme manager candidate the same short case, ideally in the room. Here is the one we use (all figures are illustrative):

The case (illustrative). A product page test ran for 14 days with a planned 50/50 split. Control: 50,410 users and 1,512 orders (3.00%). Variation: 49,050 users and 1,570 orders (3.20%). The dashboard shows a +6.7% lift and the product owner wants to ship. You have 45 minutes to decide what you would recommend and to write a five-line summary for the product owner.

A strong candidate checks trust before effect. The split is 50.7% to 49.3% instead of 50/50. A chi-square test on 99,460 users gives p ≈ 0.00002, far below the usual alert threshold of 0.001, so this is a sample ratio mismatch: something in the set-up excluded or double-counted users, and the result cannot be trusted as it stands. Even ignoring the mismatch, the difference is not significant (two-sided p ≈ 0.07). The recommendation is to investigate the cause (redirects, bot filtering, a variation that fails to load on some browsers), fix it and rerun, not to ship.

A weaker candidate reports "+6.7%, nearly significant, ship it". That is the answer the exercise is designed to catch. If you need a refresher on why, 10 A/B testing mistakes that fake a result and the A/B test QA checklist cover both traps.

Section 7 · Training the business

Train every audience for the decision it makes, not for the tool

A team of five cannot run a company's experimentation on its own, and should not try. The goal of training is to let more people take part safely: executives who read results correctly, product owners who write testable hypotheses, developers who build robust variations. Each audience needs something different, and most need far less time than a generic "A/B testing course".

What this shows. The time each audience needs is short; the point is to tie it to what they decide. The orange row is the one that scales a programme: champions who complete a course and three reviewed tests earn the right to launch tests on their own. The side panels show why it is worth the effort: Airbnb reported that team-specific Data University training cut ad hoc data requests from those teams by 50%.

What each audience needs

• Executives and sponsors (1 hour). How to read a result and its interval, why most tests lose, what a guardrail is, and the three questions to ask before accepting a win. The aim is that they reward learning, not just wins. How to talk about experimentation to every stakeholder gives the language that lands with each audience.
• Product owners and managers (half a day, then two coached tests). Turning evidence into a hypothesis, choosing one primary metric, reading results and deciding. From customer insight to test backlog is good pre-reading.
• Designers and UX writers (half a day). Designing variations that isolate one change, accessibility and brand guardrails, and when a change should simply ship without a test.
• Marketing, trading and CRM (2 hours). Why a campaign launched mid-test can ruin it, how to read a result, and how to submit ideas with evidence.
• Developers (1 day, hands-on). Tool set-up, stable selectors, flicker and performance, QA across devices, and the sample ratio check. Server-side testing for those who need it.
• Analysts (2 to 3 days, then peer review). Power and sample size, fixed-horizon versus sequential testing, variance reduction (CUPED), multiple comparisons and false positive risk. The A/B testing statistics guide covers the models.

How Booking.com and Airbnb spread the skills

Two published examples show what works at scale. Booking.com was running some 25,000 tests a year, Harvard Business School professor Stefan Thomke wrote in 2020, and he lists "any employee can launch tests" among the conditions of a culture of experimentation. To keep quality high across around 1,000 parallel experiments, Booking chose education over rigid enforcement: "Booking.com has always emphasized individual responsibility. A flexible system is better suited to our changing business needs." It relies on three community formats: Experiment Ambassadors (a network of enthusiasts, experts and data scientists embedded in product teams), Experiment Reviews (small groups review randomly chosen experiments, covering hypothesis, set-up and decision) and Experiment Bashes (regular meetings where decision-makers review past experiments). Combined with tooling such as automatic power calculations, Booking reports that the share of high-quality experiments has steadily risen. For the wider picture of how such companies run thousands of tests, see how Booking.com, Microsoft, Netflix and others experiment at scale.

Airbnb's Data University is the other model: an internal academy taught by 55 volunteer faculty from data science and engineering, with more than 400 courses taught and over 6,000 registrations since 2016 at three levels. Its team-specific "Data U Intensive" format, two to three days built on the team's own dashboards and data, cut ad hoc data requests from supported teams by 50% and was followed by a 30% increase in SQL activity among participating teams, sustained through late 2018. Data University is about data literacy rather than experimentation, but the design transfers well: volunteer teachers from the expert team, levelled courses, and training on the learner's own data.

For leaders. Training is also a retention tool. In LinkedIn's 2025 Workplace Learning Report, 88% of organisations say they are concerned about employee retention, and providing learning opportunities is respondents' number one retention strategy. The people you train are the people you keep.

Section 8 · Certification

An internal academy works best when training earns the right to launch tests

Training without consequences fades. The programmes that scale tie training to permissions: you complete a course and a few supervised tests, then you may launch tests on your own within guardrails. We call it a "licence to test". It does not slow people down. It replaces case-by-case approvals with a one-off investment in skills.

Three practical rules make the academy work:

• Teach with your own data. Airbnb's best results came from team-specific training on the team's own dashboards. A course built on your site, your tests and your mistakes is worth three generic ones.
• Make the experts teach. Booking's ambassadors and Airbnb's volunteer faculty are practitioners from the expert teams. Put teaching in your senior analysts' objectives.
• Review a random sample. Booking's Experiment Reviews pick experiments at random. That keeps quality visible without making every test wait for approval. Track the share of tests that pass QA and sample ratio checks as a programme health metric; how to measure an experimentation culture shows how.

You do not need to write every course yourself. Disclosure: Henkan & Partners runs the Henkan Academy. Our first course, A/B Testing Fundamentals, takes a learner through one test from customer insight to a documented decision in nine lessons (about nine hours), with a final exam that requires 80% to pass. Teams can use it for level 2 and add modules on their own stack.

Section 9 · AI

AI changes the skills mix of the team, not the need for judgement

AI is already part of how tests are built and analysed. Asked in Ascend2's 2025 survey how the adoption of AI-driven tools had affected their A/B testing processes, 46% of respondents said significantly improved and another 46% said somewhat improved. Optimizely says that 19.54% of follow-up tests are driven by agent recommendations grounded in prior results, in the context of its Opal agents (vendor data). Testing vendors now ship agents that draft hypotheses, generate variation code from a prompt and summarise results; AI agents in experimentation maps what each step can and cannot hand over.

For hiring and training, this has three consequences:

• Build skills shift from typing code to reviewing it. A developer who can review AI-generated variation code for selectors, performance and accessibility is worth more than one who only writes it. Add code review of AI output to developer training.
• Analysis gets faster, so judgement matters more. An AI summary of a result is only as good as the checks behind it. The analyst's job moves towards trust checks, guardrails and the decision, which is exactly what the practical exercise in Section 6 tests.
• Everyone needs AI literacy, and some need AI governance. The WEF ranks AI and big data as the fastest-growing skills to 2030. Add a module on what not to put into AI tools (customer data, unreleased prices), how to check AI output, and which uses your legal team has approved. Tools exposed to AI through MCP (Model Context Protocol, a standard way for AI assistants to call tools and data sources) need the same access rules as any analyst.

Our view. Do not cut the analyst to pay for an AI tool. AI raises the number of tests a small team can build and read. It does not tell you that a 50.7/49.3 split invalidates a result unless someone has taught it, and someone checks.

Section 10 · Careers

Career paths keep the people you trained from leaving

Experimentation specialists are easy to poach once trained, and the field is young: in CXL's 2020 survey, finding optimisers with six or more years' experience was described as a continuing challenge. A visible career path is the cheapest retention tool you have. Offer two ladders, so senior experts do not have to become managers to progress.

Section 11 · What to do next

Build the team one capability at a time, starting with the gap that blocks you

1. Map the skills you already have

Score your current team and regular contributors against the eight skills in Exhibit 3. The gap that blocks the most tests is your first hire or first contract, not the role you assumed you needed.

2. Decide hire, buy or borrow for each capability

Run each missing capability through the decision tree in Exhibit 5. Write down what you will own whatever you buy: the backlog, the repository, metric definitions and the decision on what ships.

3. Write a structured interview with a practical exercise

Use the questions in Table 3 and the 45-minute case in Section 6, with the scoring grid in Table 4. Use the same case for every candidate so you can compare them fairly.

4. Launch a training plan by audience

Start with the one-hour executive session and the product owner course, because those two audiences decide what ships. To save time, download the Experimentation Culture Kit: it includes a one-page charter, ritual agendas, a stakeholder map, objection cards, a RACI, a health scorecard and a maturity self-assessment you can use alongside the training.

5. Introduce a licence to test and a random review

Define the four levels in Table 5, review a random sample of tests each month and publish the share that pass QA and sample ratio checks. Promote the people who teach.

6. Get help where you cannot hire yet

If you need senior skills before you can hire them, a fractional team can run the programme and train your people while you recruit, with a handover plan from day one. Talk to us.

FAQ

Frequently asked questions about building an experimentation team

Frequently asked questions

What is an experimentation team?

An experimentation team is the group of people who plan, build, run, analyse and decide on controlled experiments such as A/B tests. It typically includes an experimentation lead, a programme manager, an analyst, a developer, a UX researcher, a designer and a product owner, though in small programmes one person often covers several roles.

What roles are in a CRO team?

A complete CRO or experimentation team covers seven roles: experimentation lead (head of CRO), programme manager, analyst or statistician, developer, UX researcher, designer and product owner. Champions in other teams extend its reach. Early on, the lead usually doubles as programme manager and the analyst may also set up the tool.

How big should an experimentation team be?

It depends on maturity and traffic. In our experience, a programme running fewer than 10 tests a year needs one or two people with agency or fractional support; 10 to 99 tests a year needs a core of three to five; 100 to 999 needs six to twelve in a centre of excellence plus champions in product teams.

Who should be the first hire in an experimentation team?

After the lead, hire whichever of analyst or developer is scarcer in your organisation. Without an analyst, results cannot be trusted; without a developer, tests cannot be built. Buy the other skill by the project or from a fractional team until volume justifies a hire.

Should I outsource A/B testing or build an in-house team?

Decide capability by capability. Keep the lead, product ownership and customer knowledge in-house. Buy occasional specialist work, such as audits or server-side set-up, by the project. Use a fractional team when you need senior skills now but cannot hire them yet. Always own the backlog, the test repository and metric definitions.

What skills does a CRO specialist need?

The core skills are experiment design and statistics literacy, analytics and tracking, research and Voice of Customer, enough front-end knowledge to brief and QA a variation, and clear communication. Comfort with being wrong and analytical thinking matter more than any single tool.

What interview questions should I ask a CRO or experimentation candidate?

Ask how they decide how long a test runs, what would make them distrust a significant result, and about a losing test that changed a decision. Then give a short practical case with a flawed result, such as a hidden sample ratio mismatch, and ask for a five-line recommendation.

How do you train a company in experimentation?

Train each audience for the decision it makes: one hour for executives, half a day plus coached tests for product owners, half a day for designers, a hands-on day for developers and two to three days of statistics for analysts. Tie training to a licence to launch tests and review a random sample of tests.

Key terms

Experimentation team

The people who plan, build, run, analyse and decide on controlled experiments. Defining it by roles rather than headcount lets a small programme cover every skill.

Experimentation lead (Head of CRO)

The person who owns the programme's strategy, standards and roadmap. It matters because the lead sets whether the programme rewards learning or only wins.

Programme manager

The role that runs intake, prioritisation, rituals and the test repository. It keeps tests flowing and makes results findable later.

Experiment owner

The person accountable for a specific test, from hypothesis to decision. Training more experiment owners is how a programme scales beyond the central team.

Champion

A person in another team who promotes experimentation and runs tests within guardrails. Champions extend the programme's reach without adding headcount.

Centre of excellence

A central team that sets standards, trains and reviews while product teams run their own tests. It is the usual structure once a programme runs hundreds of tests a year.

Fractional team

Senior specialists who work for a company part-time under a retainer. It gives access to scarce skills before a company can hire them, provided a handover is planned.

Licence to test

A permission to launch tests independently, earned through training and reviewed tests. It replaces case-by-case approvals with a one-off investment in skills.

Power calculation

The calculation of how many users a test needs to detect a given effect reliably. Without it, tests are stopped too early or run without a chance of a conclusive result.

Sample ratio mismatch (SRM)

A traffic split that differs from the planned split by more than chance allows. It signals a technical problem, so the result cannot be trusted until the cause is fixed.

Experimentation Evolution Model

Microsoft's model of how experimentation matures through Crawl, Walk, Run and Fly phases. It is useful for sizing a team and planning what to build next.

Skills matrix

A grid of roles against skills, with the level each role needs. It shows gaps in a team before you write a job description.

Peer review of experiments

A review of a test's hypothesis, set-up and decision by colleagues, often on a random sample. It maintains quality without slowing every test.

CUPED

A variance reduction method that uses users' behaviour before a test to reduce noise. It lets tests reach a result with fewer users, and is a common topic in analyst training.

Sources

All sources were checked in October 2026. Optimizely figures are vendor data from its own customer base; Speero and CXL figures are agency or practitioner survey data (Speero's 2025 benchmark is a self-assessment by 154 respondents, sponsored by VWO, Kameleoon and Convert); Ascend2 figures come from a survey of 402 marketing decision-makers conducted in partnership with marketing technology providers; LinkedIn figures come from LinkedIn's own survey of learning and HR professionals. Pay and job-growth figures are US Bureau of Labor Statistics data for the United States only and cover official occupations, not CRO job titles. Exhibits 2, 3, 5 and 7 (main table), the team rows of Exhibit 4 and Tables 1 to 6 are Henkan & Partners frameworks based on our experience. The interview case in Section 6 is illustrative; its statistics were calculated by Henkan & Partners.

1. Ascend2 (2025). A/B Testing in Marketing research report.
2. Speero (2025). Experimentation Maturity Benchmark Report 2025, Methods & Process (agency data).
3. Hennekes, B., CXL / Speero (2020). The 2020 State of Conversion Optimization Report · PDF.
4. Cano, E. et al., Booking.com Data Science (2026). Scaling Experimentation Quality at Booking.com.
5. Thomke, S., Harvard Business Review (2020). Building a Culture of Experimentation.
6. Fabijan, A., Dmitriev, P., Olsson, H. H. & Bosch, J. (2017). The Evolution of Continuous Experimentation in Software Product Development. ICSE 2017.
7. Kohavi, R., Tang, D. & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing, chapter 4, Experimentation Platform and Culture. Cambridge University Press.
8. Tingley, M. et al., Netflix Technology Blog (2022). Experimentation is a major focus of Data Science across Netflix.
9. Stober, J., Airbnb Engineering & Data Science (2018). How Airbnb is Boosting Data Literacy with Data U Intensive Training.
10. World Economic Forum (2025). The Future of Jobs Report 2025, Skills outlook.
11. LinkedIn Learning (2025). 2025 Workplace Learning Report.
12. US Bureau of Labor Statistics (2026). Occupational Outlook Handbook: Data Scientists.
13. US Bureau of Labor Statistics (2026). Occupational Outlook Handbook: Software Developers, Quality Assurance Analysts, and Testers.
14. US Bureau of Labor Statistics (2026). Occupational Outlook Handbook: Web Developers and Digital Designers.
15. US Bureau of Labor Statistics (2026). Occupational Outlook Handbook: Market Research Analysts.
16. Optimizely (n.d.). Top 10 takeaways from running 127,000 experiments (vendor data).
17. Henkan & Partners. Illustrative interview case: sample ratio chi-square test and two-proportion z-test, calculated October 2026.