Selecting Users For A/B (Champion/Challenger) Testing

Posted by Gordon Guthrie on Stack Overflow See other posts from Stack Overflow or by Gordon Guthrie
Published on 2010-02-22T13:15:28Z Indexed on 2010/04/19 23:33 UTC
Read the original article Hit count: 584

Filed under:

ab-testing

|

web

|

web-application

|

web-development

|

web-design

We have a framework that offers A/B split testing.

You have a 'champion' version of a page and you develop a 'challenger' version of it.

Then you run the website and allocate some of your users the champion and some the challenger and measure their different responses.

If the challenger is better than the champion at achieving your metrics then you dethrone the champion and develop a new challenger...

My question is what mechanisms should I consider to allocate the versions?

A number of options spring to mind:

odd or even IP address (or sub-segments) ****.****.****.123 gets champion but .124 gets challenger
cookie push - check for a champion/challenger cookie, if it doesn't exist randomly allocate the user to one and push the cookie

Best practice? suggestions? comments? experience?

© Stack Overflow or respective owner

Related posts about ab-testing

Nginx A/B testing

as seen on Stack Overflow - Search for 'Stack Overflow'
Hey, I'm trying to do A/B testing and I'm using Nginx fo this purpose. My Nginx config file looks like this: events { worker_connections 1024; } error_log /usr/local/experiments/apps/reddit_test/error.log notice; http { rewrite_log on; server { listen 8081; access_log… >>> More
Selecting Users For A/B (Champion/Challenger) Testing

as seen on Stack Overflow - Search for 'Stack Overflow'
We have a framework that offers A/B split testing. You have a 'champion' version of a page and you develop a 'challenger' version of it. Then you run the website and allocate some of your users the champion and some the challenger and measure their different responses. If the challenger is better… >>> More
How does Google index our site when you're doing a Website Optimizer experiment?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm about to use Google's Website Optimizer to do a/b testing on the home page of my site. My question is: which of the alternative pages will google's spider index? All of them? I couldn't find any info about this on google or on GWO pages. >>> More
Any thoughts on A/B testing in Django based project?

as seen on Stack Overflow - Search for 'Stack Overflow'
We just now started doing the A/B testing for our Django based project. Can I get some information on best practices or useful insights about this A/B testing. Ideally each new testing page will be differentiated with a single parameter(just like Gmail). mysite.com/?ui=2 should give a different… >>> More
Google Web Optimizer -- How long until winning combination?

as seen on Stack Overflow - Search for 'Stack Overflow'
I've had an A/B Test running in Google Web Optimizer for six weeks now, and there's still no end in sight. Google is still saying: "We have not gathered enough data yet to show any significant results. When we collect more data we should be able to show you a winning combination." Is there any way… >>> More

Related posts about web

Why is Java EE 6 better than Spring ?

as seen on Oracle Blogs - Search for 'Oracle Blogs'
Java EE 6 was released over 2 years ago and now there are 14 compliant application servers. In all my talks around the world, a question that is frequently asked is Why should I use Java EE 6 instead of Spring ? There are already several blogs covering that topic: Java EE… >>> More
Hosting a website on Heroku.... I know how to, but im running into problems!

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I'm starting to learn more on the back-end scale of programing. Recently I started up Heroku for the second or third time. This time I actually installed the Git update to my Mac and installed Heroku in the terminal. I wanted to upload a static html site with the sinatra gem. Everything worked out… >>> More
Microsoft .NET Web Programming: Web Sites versus Web Applications

as seen on Samir ASP.NET with C# Technology - Search for 'Samir ASP.NET with C# Technology'
In .NET 2.0, Microsoft introduced the Web Site. This was the default way to create a web Project in Visual Studio 2005. In Visual Studio 2008, the Web Application has been restored as the default web Project in Visual Studio/.NET 3.x The Web Site is a file/folder based Project structure. It… >>> More
VS2008 - Unable to Add Web Reference to Web Application Project (The web services enumeration compon

as seen on Stack Overflow - Search for 'Stack Overflow'
I've run into a situation where I was unable to add a Web Reference in Visual Studio 2008 to a Web Application Project. The error I couldn't resolve was "The web services enumeration components are not available. You need to reinstall Visual Studio to add web references to your application." How… >>> More
Outlook Web Access: "Outlook Web Access has encountered a Web browsing error"

as seen on Super User - Search for 'Super User'
When one of my colleagues is accessing Outlook Web Access from IE, he frequently gets an error reported: "Outlook Web Access has encountered a Web browsing error". The error report includes the following: Client Information User Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0; Trident/4… >>> More