What is an experiment?
In the digital world, the classic experiment is an AB test. But there are more experiment methods than just an AB test. 为了衡量人们对一个新想法的兴趣，你可以对它进行实地测试. To understand how loyal customers will respond to a new feature, you could beta test it (closed or open). If you’re releasing changes to a website, 作为发布过程的一部分，您可能会有阶段性的推出或特性标志, a type of AB test).
关键是:把实验想象成一种测量变化效果的方法. AB tests, beta invites, fake doors, all of the above.
所以，现在我们清楚了有多种类型的实验，让我们回到这个. 实验有两个概念——验证性实验和探索性实验.
我将尽我所能解释这两个概念，并给出一些优缺点. Strap in…
Confirmatory experiment
我从验证性实验开始，因为这是你们可能熟悉的实验类型.
想象一下，你刚刚观看了一天的可用性测试，10个用户中有7个访问了你的产品页面，却错过了免费送货信息. 从之前的调查中你知道免费送货对你的客户很重要.
您决定需要在产品页面上更明显地显示免费送货信息. 你让你的设计师设计一个新版本的产品详细信息部分，这次, make free delivery more obvious!
Your hypothesis might go like this:
因为我们观察到，在可用性测试中，有7/10的客户错过了产品页面上的免费送货信息, while surveyed customers indicated free delivery is important.
We believe making the free delivery message more prominent.
这将导致更多的客户看到信息，并知道我们的免费送货, therefore being more likely to purchase.
Confirmed when purchase rate from product pages increases.
你的设计师开始工作，制作了一个可爱的新设计，让免费送货真的……很明显.
You test it. It wins. 8% increase in purchase rate. 96% significance. Bosh. For the love of data, someone please crack open the champagne!
This was a confirmatory experiment.
You theorised that version B (bigger, bolder free delivery) was better than version A (puny, diminutive free delivery). You tested that theory. You’ll learn one of three things from this experiment:
1. Version A is better (damn )
2. Version B is better (yay )
3. Versions A and B are the same (meh )
Now let’s look at an alternative scenario.
Exploratory experiment
We’re back to the end of the day, 你刚刚看到所有人都错过了免费送货信息. You know it’s a vital message.
Your hypothesis is the same. However, this time you give your designer the following brief:
- People are missing the free delivery message on product pages, but we know it’s important for them to see it.
- Can you give me a few variations of the message? 首先，尝试在用户旅程的早期让信息更加突出的设计. 让我们在产品结果页面的第二行结果之后放置一个横幅.
- Next, try making it more prominent on the product pages. Put it somewhere near the price.
- Finally, add a free delivery message to the basket page. 把它显示在总价附近，这样就很清楚以后不会加进去.
你的设计师设计了三个不同的版本，我们称之为变体. You run an AB/CD test with all three against the current site.
Here’s what you might learn from this experiment:
1. A is best, then B, C, D
2. B is best, then A, C, D
3. B is best, then C, A, D
4. D is best, A, B, and C are equal
5. C is best. B and D are equal. A is worst.
6. And so on…
我的意思是，你从这个实验中学习的能力大大提高了. 通过一些聪明的分析，您可以了解到在旅程中更早地显示信息更好. You could learn the exact opposite: the later the better. 你可能会发现弹出窗口有负面影响——可能是因为它是一个令人讨厌的弹出窗口——而改进的产品页面横幅有积极影响(8%), remember?) but that the basket message has a HUGE positive impact (14%!).
您甚至可以重复这个实验，并同时运行产品页面的版本2 and basket page banners. You might find that combined you get an even greater uplift of 16%!
The key thing is: you learned.
“Holy crap I’m sold.”
What’s the catch? Nothing big. But there are some considerations.
You must have enough traffic to flex
Google “ab sample size calculator” and use one of the many free tools. If you need 10,000 sessions per variation and you get 10,你可能想要探索可用性测试并通过AB进行确认.
Don’t throw designs at the wall and hope they stick
这就像你第一次学习什么是多变量检验并做21变量分裂检验一样.
相反，你应该清楚地表达你想教给你的设计师(或者你自己，如果你是你的设计师)什么。. Think about measuring on a spectrum. Examples include: early vs late in the journey; subtle vs LOUD; emotional vs rational; price-driven vs quality-driven.
你希望能够从这个(一系列)实验中解脱出来，并告诉你的团队:
“我们在旅途中用不同的方式测试了免费送货信息，我们发现最好在提到价格的地方提供免费送货信息, and to include it in a subtle way rather than an obnoxious banner.”
Final thoughts
Don’t think I’m saying confirmatory experiments are bad! They totally have a place. Sometimes it’s not feasible to design and build three or four variations. 有时你需要提出问题，最好通过可用性研究来探索. 有时你只有一个清晰的想法，你想看看是否更好地说出其他的想法.
If you take anything from this post, 而是你应该知道两者的区别，并寻找机会去探索. If it’s not feasible this time, try again next time.
Keep experimenting.
