Some of the examples in this chapter compared different rates (bounce, exit, conversion, etc.) across traffic sources and media. We should note that comparing different rates or percentages can be a little deceptive. The fact that one particular rate outperforms another is not enough unless we determine whether the difference is statistically significant. Statistical significance exists when a difference between two rates is large enough that we can attribute it to something other than an expected sampling error. Consider the following example of comparing conversions among different engines:

It looks as though traffic from Yahoo! is converting better than traffic from Google. A quick calculation tells us that Yahoo! traffic provides about 10.50% uplift in conversion. The question is whether the difference is large enough to determine whether we have a significant improvement from Yahoo!. The actual mathematical formulas used to determine statistical significance are beyond the scope of this book—but no worries, you do not have to learn these formulas. Statistical significance calculators are available online that you can use to determine the validity of your conclusion.

Figure 2-10 shows that the difference in conversion rate between Google and Yahoo!—when evaluated using a statistical significance calculator—falls within the margins of error and is not noteworthy. Figure 2-11, on the other hand, shows that the difference in conversion rates between Bing and Yahoo! is statistically significant.

Figure 2-10. Statistical significance between Yahoo! and Google

Figure 2-11. Statistical significance between Yahoo! and Bing