Our discussion has focused on quantitative testing with measurable results. Though it goes beyond the scope of conversion optimization, you should not ignore the value of usability testing to see how visitors interact with your website. Designing a good usability test can be tricky. Consider the following when designing your usability tests:

  • Create a list of different tasks you want your visitors to complete on your website. These tasks should vary in complexity and test different functions of the website.

  • Record users’ interaction with your website during usability tests. You can always go back and review how people reacted to the site.

  • Interview users after a test is completed and ask them about the different tasks they had to perform.

Tip

DON’T FORGET!

CASE STUDY: Increasing Account Sign-Ups at YouTube

YouTube is the global leader in online video sharing and is the third most trafficked site in the world (Alexa, August 2010), attracting approximately 102 million unique visitors per month (Compete.com).

YouTube needed to increase account sign-ups from its home page because:

  • YouTube relies on fresh video content being constantly uploaded by users. A user must have an account to upload a video.

  • Once a user signs up for an account, YouTube is able to provide the user with added site functionality, such as video playlists, favorites, and so on. This additional functionality helps to increase both user engagement and user retention.

  • Crucially, YouTube is able to gather information about account holders during the sign-up process. In addition to this, as members interact with the site over time, their behavior and interests provide further information. This insight into its members is critical in enabling YouTube to fully leverage and target its on-site advertising.

  • Finally, YouTube intends to increase the amount of premium content that can be purchased on the site. User accounts are required for premium content.

While devising a plan to increase the conversion rate, the optimization team that worked on this went through the following steps.

Step 1: Researching

The team started by researching the types of visitors using the test page (home page) and what they were looking for. The goal was to understand the gap between user intent and page purpose. For example, if a user lands on the home page and is looking for a video, what benefits should the page offer to get the user to change course and sign up for an account instead? If visitors land on the home page and want to create an account, how could the page help them achieve this goal?

Step 2: Developing Hypotheses

For the home page, the team had three main hypotheses:

Step 3: Testing the Plan

The optimization team used the test hypotheses as the basis for the test plan. The test plan outlines the test structure, test goal, test hypotheses, and test elements to compete against the original page and validate the test hypotheses.

Figure 9-19 shows the three elements on the YouTube home page that related to account sign-ups. Two of these elements were in the header banner, and one was located in the righthand column.

The large volume of traffic that YouTube receives enabled the team to devise an unusually massive multivariate test with 1,024 variations:

Section 1: 2 variations
Section 2: 16 variations
Section 3: 32 variations
Total: 2 * 16 * 32 = 1,024 combinations

This was a very large test—not typically recommended for most sites. But given the massive traffic levels YouTube generates, the sample size was sufficiently large to reach statistical significance quickly. This was the largest Google Website Optimizer test run to date (in terms of traffic volume), and statistical significance was attained within a day or two.

Figure 9-20 shows the test results from Google Website Optimizer.

Figure 9-21 shows the original (control) page.

Figure 9-22 shows combination 28, which came out on top, with a 15.7% increase over the control.