Generate ideas for experimentation based on direct data

  • Updated
This article is part of The Optimization Methodology series.
This topic describes how to:
  • Use direct data about visitors to your site to generate strong hypotheses
  • Consider how to use multiple data sources together

Many organizations rely on direct data, or direct intelligence about their site experience, to drive the ideation process. And no wonder—today’s optimizers have dozens of analytics platforms at their fingertips. But it can be a challenge to use all that data well.

Elite optimizers use direct data to tell a story about their visitors: who they are, what they are trying to do, and how they are trying to do it. With that knowledge, they form data-driven hypotheses about the actions a visitor is likely to take—and optimize for them.

Below is a list of the most common direct data sources:

Do not feel that you need to consult all these sources equally. Use this list to consider which can generate actionable insights to guide your experimentation. With a few different data sources, you can see a bigger picture to understand better how visitors navigate your site experience.

Use the insights you generate to build a business intelligence report, which helps support a strong experimentation program. 

Direct data (data about your visitors) is only half the picture. Read about using indirect data to tackle opportunities that your analytics surface.

 

Web analytics

Many organizations rely heavily on behavioral web analytics for ideation. Behavioral web analytics can help you understand where to optimize but not necessarily how. Pair this data with other direct and indirect data sources to decide how to optimize for different opportunities on your site.

Use case

How this adds insight

Focus your analysis on figuring out where to test. Use pathing and funneling analyses to look for disproportionate drop-off rates. Or, monitor traffic patterns and “page values”—an estimate of how much revenue a given page generates—to find the most valuable content on your site.

If you find that 20% of your site generates 80% of the value, focus your data collection and optimization efforts on that valuable real estate.

Once you have identified bottlenecks in your most valuable flows, layer on qualitative data such as surveys and heatmaps for additional insight.

Should you surface your most valuable products where visitor attention is concentrated? Is an unclear step in your sign-up flow resulting in an unusually high drop-off?

Use indirect data such as reviews of competitor sites to note different design decisions that you might test.
For a detailed walkthrough of how to use your analytics to generate experiment ideas, check out this article on using basic analytics reports or using advanced analytics reports.

Experiment results

Often, the most actionable source of quantitative data is the results of your past experiments and campaigns. The difference between the success of two experiences can tell you a lot about what your visitors want.

Use case

How this adds insight

Say you move a button to a different place on the page and this results in a higher click-through rate. You have clear evidence that button placement matters to your visitors.

Are there similar types of changes that might matter? Would changing the hero image or providing different information above the fold increase conversions?

Are some product categories more or less affected by the experience you created? Did certain media campaigns underperform?

Web analytics provide context for the results of a past experiment. Heatmaps and survey results can also shed light.

For example, a more prominent button encourages more click-throughs but results in more visitors exiting the site. Qualitative data can help you understand why and how to encourage visitors to stay and browse.

 

Voice of the customer

Surveys, NPS scores, and the experiences of your support team can reveal facets of your customer experience that you might not find in your other data.

Use case

How this adds insight

At Optimizely, we use NPS surveys to gather customer feedback on how we can improve. Over the years, these surveys have provided much of the feedback that Optimizely uses to bring products to market, such as Stats Engine.

Take a look at your NPS scores. Are there any actionable trends in that customer feedback?

Pair an NPS scoring system with different experiment variations to gather direct feedback from customers about each experience.

These responses can help you understand why a page is underperforming and where there are missed opportunities to optimize your site experience.

Due to low response rates and small sample sizes, it can be tricky to draw definitive conclusions based on survey feedback and NPS scores. Try tagging your survey responses with a topic category or keyword to find clearer trends.

Heatmaps

Heatmaps tell you where your visitor’s attention is focused and what the most important part of your page layout is.

Use case

How this adds insight

Landing pages are some of the most important places to optimize, particularly for B2B, lead generation, and consumer service businesses.

Once you understand which parts of your landing get the most attention, redesign the page to highlight the most important CTAs, headlines, or videos.

If your site has multiple landing pages, it can be difficult to reduce the bounce rate across all of them. Heatmaps show you trends; for example, if a certain element distracts visitors from your ultimate goal.

You can also run a heatmap analysis on your competitor’s landing page or an admired third-party experience to compare.

Certain tools like EyeQuant do not require a behavioral tracking code, so you can run them on any site you are interested in.

User personas

User personas distill the complex and varied characteristics of your customers into a few themes that help you make savvy marketing decisions. Interviews, surveys, analysis, and brainstorms help you create personas that inform your testing and personalization efforts.

Use case

How this adds insight

User personas are a helpful cue for framing campaigns and experiments. Use them to decide how to target headline copy to a C-suite executive, a budget shopper, or a social sharer, for example.

You do not necessarily have to develop full, formal profiles. Informal personas such as “customers who have not purchased in a long time” or “visitors who are actively comparing our services to our competitors’ services” can be just as actionable.

When combined with web analytics, user personas help you understand customers’ motivations.

For example, if certain shopping behaviors such a cart that includes kitchenware and holiday decorations indicate a customer buying for a family. What targeted message or a promotional incentive can you offer that persona?

Personas can also shed light on indirect data like competitor reviews. Does your competitor’s site and messaging target a specific persona? How does this affect how you position your own brand or design messaging for your site?

Usability tests

Usability testing shows you common pitfalls in your site experience. Conduct manual tests with live sessions or use a platform like UserTesting.com to gather data asynchronously.

Use case

How this adds insight

Usability testing helps you answer important questions such as:

  • Can users who are unfamiliar with your site successfully complete important tasks?

  • Are certain moments of complexity or uncertainty in your site experience prompting visitors to exit a funnel?

Review mockups, competitors’ sites, and new features before you take them to market.

Use your survey results, competitor reviews, and web analytics to develop prompts for usability testing.

Imagine that your web analytics indicate poor performance on a landing page and a heatmap shows that visitors are not engaging with the CTA.

Based on this information, design a user testing prompt that asks: “Tell us the #1 thing you feel this page communicates. How long did it take you understand that message? Is it compelling? What else would you like to see? What is the next step you would take on this page?”

Customer decision models

Some consumer decisions—such as buying a home, enrolling in a new health insurance plan, or researching a SaaS product—require multiple touch points before a final decision can be reached. Often, these touch points occur offline as well as online. Smaller online touchpoints are sometimes referred to as micro-conversions.

Build a model to describe the triggers and interactions a customer is involved in during the decision-making process. This can help you optimize for the micro-conversions that lead a customer to complete your key transaction.

Use case

How this adds insight

Create hypotheses that focus on the most important micro-conversions in your customer decision model.

What triggers indicate success in this model (scroll depth, video views, or broad content consumption)? What behaviors should you optimize for?

Monitor how optimizing these micro-conversions affects your leading indicators.

Look for patterns among visitors who display the behaviors that you consider micro-conversions. Combine your customer decision model with your CRM, web analytics, and personalization strategy to target CTAs and value propositions based on where a customer is in the decision-making process.

Do new visitors want to see proof that your service is reputable, before they engage with videos or demos? Conditionally show those visitors social proof and data points to encourage them to engage.

Once a visitor has seen your basic content, consider moving them deeper into the model; the visitor may now be ready to provide an email address in exchange for additional content like technical specifications.

Metrics

Often, there is an implicit tradeoff between one metric versus another. Monitor the interaction effects between different metrics to understand how an improvement to one might detract from another. You might also directly test these theories to optimize for aggregate profit.

Use case

How this adds insight

Imagine an online retailer finds that offering a promotional discount improves her conversion rates. However, this improvement is at the expense of cart size and average order value (AOV).

She might decide to experiment with those discount levels to model a price-elasticity curve. This model can help her maximize total revenue based on tradeoffs between unit margins, purchase conversion rates, and AOV.

Or, in media publishing, prominent social share buttons can reduce the likelihood that a visitor clicks through to another article. Likewise, optimizing for ad impressions above the fold can detract from scroll depth and article completion rates.

Monitor how competing conversion metrics affect important downstream goals like acquisition and retention.

Best-in-class site experiences, academic publications, and your competitors’ sites can provide a sense of how far you can push the envelope when optimizing for different metrics.

These sources help you understand how other players in your space negotiate the tradeoffs. For example, research shows that business travelers are less likely to sacrifice details in their travel itineraries for a lower price than leisure travelers. For travel providers, this industry-level insight provides context for how to optimize for different metrics.

Web analytics and heatmaps also help you design experiments that test different metrics. For example, the exit rate at different stages of your funnel can indicate when to introduce a promotional banner into your visitor’s experience.
There are many direct data sources you can draw from for ideation. Over time, your direct data will help you develop a deeper understanding of your customers so you can test meaningful hypotheses and a robust optimization roadmap.