Use a business intelligence report to ask the right questions

  • Updated
  • Optimizely Web Experimentation
  • Optimizely Performance Edge
  • Optimizely Feature Experimentation
  • Optimizely Full Stack (Legacy)
This article is part of The Optimization Methodology series.

Ideation, the process of generating meaningful hypotheses to drive experimentation, is one of the most important steps in experience optimization. It is also one of the hardest. Optimizely Experimentation's 2015 Benchmark Report suggests that optimization programs face three hurdles:

  • Companies over-rely on quantitative and behavioral data to generate test ideas

  • Most organizations use too few data sources as inputs for their hypotheses

  • Teams do not build a consistent process for consulting the data sources they use

Without a structured framework supporting the creative stage of experience optimization, it can be difficult to consistently generate impactful ideas. A business intelligence report helps you build that structure and consistency. It gathers key insights about your site and industry into a single document. When you brainstorm hypotheses, the report helps foster creativity and ground your team's idea generation in data.

Use insights from your report to generate ideas for experimentation. An intelligence report rich in stories about your customer journey helps you plan meaningful, effective experiments.

Materials to prepare

  • Direct data – Analytics, voice of the customer, heatmaps, user testing, and more.
  • Indirect data – Screenshots of competitor sites, articles, consultant recommendations.
  • Goal Tree – Alignment with business goal. 

People and Resources

  • Program Manager
  • Design resources
  • Marketer
  • Analysts

Setup

  • Collect data for the intelligence report
  • Analyze your data
  • Create a report

What you end up with

  • A business intelligence report

Watch out for

  • Remembering to perform this process year-round
  • Neglecting to use this report for decision-making about the site in general, instead of just for ideation

This article is a part of the Optimization Methodology series.

What to include

Direct data, indirect data, and goal trees all provide different perspectives on your site experience.

experiment ideas

  • Direct data (your own site) – Web analytics, results from prior experiments, voice of the customer, heat maps, user testing, user personas, customer decision models, lifetime value (LTV), and customer acquisition cost (CAC).
  • Indirect data (industry insights) – Competitors' sites, best-in-class experiences, blogs, webinars, user groups, consultations, books, academic research.
  • Goal maps – Align to your company goals, so your experiments improve metrics that matter.

Together, these information types provide insights beyond what you might see in a single source.

For example, imagine that your web analytics report (direct data) reveals that visitors drop out of the funnel on a certain page. As a result, you may decide to test a clearer CTA to push visitors further down the funnel. But in the process of reviewing competitors' sites (indirect data), you notice another site personalizing their CTA. After reading about this idea on an industry blog, you decide that this tactic has the potential to be highly impactful. At this point, you weigh your options and decide to test whether personalizing the CTA works with your visitors.

In this example, the direct source (web analytics) tells you where to optimize. Indirect sources complement this data by suggesting how you might approach this opportunity. When used together, different types of sources shed light on questions surfaced by another data source.

Build the report

The comprehensiveness and depth of the intelligence report depends on your program’s resources and maturity level. But any program, no matter the size, can use the report to strengthen its ideation process.

Here is an overview:

  • When will you produce your report? – Produce a timeline with milestones. Plan to finalize the report before your next major planning session.

  • What data sources do you use? – Identify the sources that your organization uses. Use the list above to identify new potential resources by focusing on sources that are meaningful for your business and that you can feasibly incorporate given team constraints.

  • Who are your stakeholders for this report? – Assign responsibility for different aspects of the report to internal and external stakeholders. For example, ask the marketing team to review value propositions on competing sites or the web analyst to provide funnel segmentation analysis.

  • What are the key insights did you find? – Identify a few from each data source to include in the report. Summarize these insights and gather them into a single deliverable.

  • How will you use your report? – Introduce the report and socialize its use at the beginning of your planning meetings. Use it to generate ideas for experimentation and keep revising and reviewing it at a regular cadence.

Best practices

Review at a regular cadence

Prepare your business intelligence report so it is ready for review at major team planning and decision-making sessions. Do this bi-weekly, quarterly, or monthly, depending on how often your team meets for ideation.

review at a regular cadence

Focus on stories and insights

To get the most out of your report, fill it with it with meaningful insights and data that has a story to tell about your visitors. An intelligence review should feed your team's creative process; use it to bridge the gap between enriched events export data and a well-designed hypothesis.

Use takeaways, summaries, and quick insights to tell stories about your customers' desires, optimization opportunities, and competitive advantages. Leave out metrics that are more useful for performance reports about KPIs and revenue; for purposes of ideation, this information can be a distraction.

By using an intelligence report to drive a structured ideation process, you lay the groundwork for a successful, sustainable program.