Site redesigns: Test through the design process

Aim for the global versus local maximum

Why test through a major redesign instead of waiting until after the redesign is done? Because this lets you test the global maximum instead of many local maxima.

A/B Testing: The Most Powerful Way to Turn Clicks Into Customers, written by Optimizely Experimentation's co-founders Dan Siroker and Pete Koomen with Cara Harshman, describes the global and local maximum this way:

Imagine you’re climbing a mountain: if your goal is to get to the top of the tallest mountain, and you don’t have a map of the range, it’s probably not a good idea just to start walking up the nearest slope. You’ll climb, and climb, and then ultimately reach some peak—and then what? Where do you move next if this peak doesn’t turn out to be the highest one?

In optimization, the term for the nearby, uphill peak is the local maximum, whereas the distant, largest peak is the global maximum.

By thinking and testing big changes, you will see statistically significant results more quickly, and you will be more likely to find optimal experiences. Site-wide redesigns are perhaps the best time to test global, big-picture changes. For example, test out completely different navigation bars or product categories.

Making bigger changes helps you explore potentially winning possibilities, rather than just refine a decision that has already been made.

For some businesses, this is not an easy change to make; however, businesses that continually refine and tweak will most likely see diminishing returns on their optimization program compared to those with a willingness to explore.

Data-Driven Redesigns

For many organizations, sitewide redesigns are based on opinions: from executives, from design firms, and from other internal stakeholders.

When opinions and gut feelings drive the redesign process, you are unlikely to discover the solution that actually provides the best user experience or drives the most conversions.

Why? It is simple: opinion-driven redesigns solve the problems of the people who are giving their opinions. You are solving your stakeholder's problem, not your user's. They also encourage you to make big changes that do not actually solve user experience issues or potentially make the user experience worse.

As you redesign, get qualitative feedback from your users through user tests, focus groups, and beta testing. But do not stop there—pair the qualitative feedback with quantitative data generated through A/B tests. These techniques will give you a sense of the real issues affecting customers, and which changes will address those issues.

For each test you run, the qualitative feedback that you collect should drive the hypotheses that you test. Using feedback-based hypotheses will help you explore ideas systematically, then develop the solutions that work, not just the ones that sound cool.

Test Component by Component

So you have decided to test through your redesign, instead of rolling out your new experience based on pure opinions. Companies often wonder whether to roll out two dramatically different experiences and test them against each other, or test component-by-component changes until they have a new experience.

Testing entire site designs at once helps you explore dramatically different concepts, but keep in mind that this approach can be affected by visitor bias. Rolling out a completely different experience all at once can be shocking and uncomfortable for your visitors. Chances are that they will react more negatively than they would have otherwise.

Because visitors typically do not like change, they will generally react negatively during a broad redesign.

Testing big-picture ideas in phases, rather than all at once, can help isolate the impact of each change and minimize the perceived change to your visitors. A more advanced testing program may then test combinations of these changes with a multivariate experiment to see the interaction effects. Finally, you can A/B test entirely different experiences against one another to see the total effect.

Because you will have data on each component, you can use the insights from these smaller tests to drive your overall redesign.

In Optimizely Experimentation, your re-design process might follow this pattern:

• Prioritize the areas of your site that should be redesigned, based on the areas where you see the most user feedback or friction. Remember, not everything needs to be redesigned.
• Start with simpler A/B tests on components of your site. If you are working with the global navigation or other changes that affect multiple changes, use substring match URL targeting in Optimizely Web Experimentation to apply the change across multiple pages.
• Move to multivariate tests to analyze combinations of changes, or multi-page tests to analyze the effect of different changes along a multi-page flow or funnel.
• Finally, run redirect experiments that present completely different experiences to visitors.