- Optimizely Web Experimentation
- Optimizely Performance Edge
- Optimizely Feature Experimentation
- Optimizely Full Stack (Legacy)
Ideation, the process of generating meaningful hypotheses to drive experimentation, is one of the most important steps in experience optimization. It is also one of the hardest. Optimizely Experimentation's 2015 Benchmark Report suggests that optimization programs face three hurdles:
Companies over-rely on quantitative and behavioral data to generate test ideas
Most organizations use too few data sources as inputs for their hypotheses
Teams do not build a consistent process for consulting the data sources they use
Without a structured framework supporting the creative stage of experience optimization, consistently generating impactful ideas can be difficult. A business intelligence report helps you build that structure and consistency. It gathers key insights about your site and industry into a single document. When you brainstorm hypotheses, the report helps foster creativity and ground your team's idea generation in data.
Use insights from your report to generate ideas for experimentation. An intelligence report rich in stories about your customer journey helps you plan meaningful, effective experiments.
Materials to prepare
- Direct data – Analytics, voice of the customer, heatmaps, user testing, and more.
- Indirect data – Screenshots of competitor sites, articles, and consultant recommendations.
- Goal Tree – Alignment with the business goal.
People and Resources
- Program Manager
- Design resources
- Collect data for the intelligence report
- Analyze your data
- Create a report
What you end up with
- A business intelligence report
Watch out for
- Remembering to perform this process year-round
- Neglecting to use this report for decision-making about the site in general instead of just for ideation
This article is a part of the Optimization Methodology series.
What to include
Direct data, indirect data, and goal trees provide different perspectives on your site experience.
- Direct data (your own site) – Web analytics, results from prior experiments, voice of the customer, heat maps, user testing, user personas, customer decision models, lifetime value (LTV), and customer acquisition cost (CAC).
- Indirect data (industry insights) – Competitors' sites, best-in-class experiences, blogs, webinars, user groups, consultations, books, and academic research.
- Goal maps – Align to your company goals so your experiments improve metrics that matter.
These information types provide insights beyond what you might see in a single source.
For example, imagine that your web analytics report (direct data) reveals that visitors drop out of the funnel on a certain page. As a result, you may decide to test a clearer CTA to push visitors further down the funnel. But in the process of reviewing competitors' sites (indirect data), you notice another site personalizing their CTA. After reading about this idea on an industry blog, you decide that this tactic has the potential to be highly impactful. At this point, you weigh your options and decide to test whether personalizing the CTA works with your visitors.
In this example, the direct source (web analytics) tells you where to optimize. Indirect sources complement this data by suggesting how you might approach this opportunity. When used together, different types of sources shed light on questions surfaced by another data source.
Build the report
The comprehensiveness and depth of the intelligence report depend on your program’s resources and maturity level. But any program, no matter the size, can use the report to strengthen its ideation process.
Here is an overview:
When will you produce your report? – Produce a timeline with milestones. Plan to finalize the report before your next major planning session.
What data sources do you use? – Identify the sources that your organization uses. Use the list above to identify potential resources by focusing on meaningful sources for your business that you can feasibly incorporate given team constraints.
Who are your stakeholders for this report? – Assign responsibility for different aspects of the report to internal and external stakeholders. For example, ask the marketing team to review value propositions on competing sites or the web analyst to provide funnel segmentation analysis.
What are the key insights did you find? – Identify a few from each data source to include in the report. Summarize these insights and gather them into a single deliverable.
How will you use your report? – Introduce the report and socialize its use at the beginning of your planning meetings. Use it to generate ideas for experimentation and keep revising and reviewing it at a regular cadence.
Review at a regular cadence
Prepare your business intelligence report so it is ready for review at major team planning and decision-making sessions. Do this bi-weekly, quarterly, or monthly, depending on how often your team meets for ideation.
Focus on stories and insights
To get the most out of your report, fill it with meaningful insights and data with a story to tell about your visitors. An intelligence review should feed your team's creative process; use it to bridge the gap between experimentation events export data and a well-designed hypothesis.
Use takeaways, summaries, and quick insights to tell stories about your customers' desires, optimization opportunities, and competitive advantages. Leave out metrics that are more useful for performance reports about KPIs and revenue; for purposes of ideation, this information can be a distraction.
By using an intelligence report to drive a structured ideation process, you lay the groundwork for a successful, sustainable program.