Are you new to Optimizely Experimentation or just getting started with experimentation? This glossary defines the terms and concepts you will find in Optimizely's Experimentation interface, articles, and Academy.
You can also check out our online Optimization Glossary, which defines and describes general optimization-industry terms.
Tip: Use Ctrl+F on PC or Command+F on a Mac to search this page for the exact term you are looking for.
| Term | Definition |
| A/A testing | Comparing the same webpage or mobile app experience against itself to test the accuracy of the testing tool. Read Run and interpret an A/A test for details. |
| A/B testing (or split testing) | Comparing one or more variations of a webpage (or pages) against each other and measuring how the two variations perform for specifically defined metrics. Read our blog on A/B testing ideas for a list of ideas. |
| A/B/n testing | Comparing multiple versions of a webpage or mobile application against each other to determine which performs best. |
| Account | Highest level of the hierarchy in your Optimizely plan. All your projects, experiments, campaigns, variations, audiences, events, and collaborators are part of your Optimizely account. |
| Administrator | Optimizely collaborator or user role with access to all features of the account, including billing. |
| Algorithm | The mathematical formula used to execute against the chosen strategy. For a given optimization strategy, we could use several possible algorithms and an infinite number of ways to tune the algorithm’s parameters. See the FDR Control with Adaptive Sequential Experimental Design white paper for technical details. |
| Audience | Group of visitors who share a characteristic, such as the way they came to your page or the browser they use. You target experiments based on audiences. |
| Average order value (AOV) | The average dollar amount spent each time a visitor places an order on a website or mobile app. AOV is a common metric for e-commerce and retail sites. |
| Body content | The main column of content on a webpage. "Body content" differs from the HTML term "body," which means ALL the content on a webpage. |
| Breadcrumb | Increasingly specific category links visitors see at the top of a webpage that indicates the categorization pathway or trail a visitor followed to the current page. An example breadcrumb trail is Home > Clothing > Mens > Jeans. For ideas for experiments that involve breadcrumbs, review 10 common experiments and how to build them in Optimizely Web Experimentation. |
| Bucket | Visitors who see a given variation are "bucketed" into that variation. Similarly, a visitor can be "bucketed" into an experiment, meaning that the visitor is eligible to see the experiment's variations. For an example of how visitor bucketing works, see this page. |
| Click event | Event that measures whether a visitor clicked a certain element on the page. |
| Collaborator roles | Level of access for users within Optimizely. Optimizely collaborator roles include Administrator, Project Owner, Editor, and Viewer. |
| Conversion | Action or event that a visitor did or did not do. Examples of conversions include making a purchase, completing a form, reaching a page, and clicking a button. |
| Conversion rate | The number of conversions on a website, webpage, or mobile app divided by the total number of visitors. On the Results page for your experiments and campaigns, you will see the conversion rate for each metric you have set. |
| Conversion rate optimization (CRO) | Using A/B testing to increase the conversion rate for a website, webpage, or mobile app. |
| Call to action (CTA) | Primary button or link on a page that you want visitors to click. |
| Content management system (CMS) | A platform that allows you to edit content and publish to a website without coding an HTML page. With CMS platforms, you can centralize data editing, publishing, and modification from a single back-end interface. You can implement Optimizely on various CMS platforms. |
| Cross-project events | Optimizely feature that helps you track events across your Optimizely Web projects. For more information about cross-project events, click here. |
| Custom attribute | An attribute of the visitors that you add through Optimizely's API. Unlike Audiences, which have binary (yes/no) values, custom attributes can have multiple values. |
| Direct entry | When a visitor enters a website's domain by typing in the exact address rather than by another means like clicking a search engine result or a referral link on a social media site. An example of a direct entry is a visitor typing "www.optimizely.com" in the URL address bar to go to Optimizely's website. |
| Editor (collaborator or user role) | Optimizely collaborator or user role with access to create variations in a specific project, but not start or pause experiments. Administrators and Project Owners can grant access to a project to Editors. |
| Editor (code) | Section of the Optimizely interface where you can edit your variations within an experiment. Optimizely has the Visual Editor and also allows you to incorporate custom code. |
| Engagement metric | Average or counting metric that tabulates to what extent a visitor made a particular action in contrast to a conversion metric, which tabulates whether a visitor did or did not do something. Example engagement metrics include average pageviews and time on site. |
| Events | The success metric by which experiments are evaluated. Reveiw the aritcle on The difference between Events and Metrics for more. |
| Exact match | URL match type that runs an experiment or campaign only when a visitor loads the exact URL you specify. |
| Experiment | Optimizely's term for an A/B, multivariate, or multi-page test. Also called a test or a campaign. |
| Extensions | Optimizely feature developers can use to create custom features for non-technical teams to use. Refer to Create reusable templates for custom features using Extensions for details. |
| Full factorial testing | Testing that runs every single possible combination as its own unique variation. Full factorial testing is different from the Taguchi method (or fractional factorial testing), which runs a limited number of variations based on the variations that are expected to be the most different from each other. The benefit of full factorial testing is there is no guesswork. The downside is that running so many variations requires a lot of traffic to run the test. |
| Funnel (or sales funnel) | The path a visitor takes through a website or mobile app as they move toward becoming a customer. |
| Headline testing | Developing multiple title variations for an article or other item published on the web to determine which title performs the best. |
| Heatmap | Report that shows where visitors click on a page. Heatmaps are usually displayed as an overlay, with more intense colors indicating more visitor clicks. |
| Hero image | Large image at the top of a webpage. Hero images are sometimes called "rotating heroes" if they are slideshows that rotate through several images. |
| Home page | Page in the Optimizely interface from which you can manage projects and experiments. |
| Implementation (or snippet implementation) | Adding the Optimizely snippet to your webpage. Check out this article for more information on implementing the snippet for Optimizely. |
| Improvement |
For A/B experiments: Optimizely displays the relative improvement in conversion rate for the variation over the baseline as a percentage for most experiments. This is true for all A/B experiment metrics, regardless of whether they are binary conversions or numeric. For example, in Optimizely Experimentation, a relative improvement interval of 1% to 10% means that the variation sees between 1% and 10% improvement over baseline. For example, if the baseline conversion rate is 25%, you can expect the variation conversion rate to fall between 25.25% and 27.5%. Refer to the Confidence intervals and Improvement intervals article for more information and how improvement intervals are calculated for other types of experiments. |
| IP address | Internet protocol address: the numerical label assigned to any device on a network. IP targeting and IP filtering are based on visitors' IP addresses. |
| JavaScript (or JS) | Computer programming language. The Optimizely snippet is written in JavaScript. You can also include custom JavaScript in an experiment or use custom JavaScript to target experiments. |
| jQuery | JavaScript library is included in the Optimizely snippet and used to create variation code. Learn more about jQuery in Optimizely. |
| Landing page optimization | Improving elements on a landing page to increase conversions. |
| Lead generation | Generating consumer interest for a product or service to turn that interest into a sale. Check out our ideas for lead generation testing. |
| Marketing technology stack | Group of technologies that a marketer uses to conduct and improve their marketing activities. |
| Metric | Quantitative measurement of a visitor action. Metrics are based on events and help you measure differences in visitor behavior based on changes you make to your site in experiment variations or personalized experiences. |
| Metrics builder | Natural-language interface for metrics in Optimizely that allows you to define metrics that more closely match your business needs, view the same event as multiple metrics, and decide the winning direction for a metric. |
| Monthly unique visitors (MUVs) | Individual visitors who access your site within 30-day window. Optimizely tracks unique visitors with a cookie and bills your account based on the number of MUVs who enter an experiment on your site. Check out this article for details about MUVs. |
| Multivariate testing (MVT) | Testing that runs multiple variations of specific elements. The experiment is defined by selecting the elements and their variations, and the testing platform automatically generates the possible testing combinations. For example, 3x2 MVT means changing 3 elements with 2 variations per element. If a full factorial test is running, this will result in 3^2 (i.e., 9) variations. |
| Multi-armed bandit |
A class of problems in probability theory where an agent must decide how to best distribute resources among several options with initially unknown payout structures. The term "multi-armed bandit" derives from a person, the “bandit,” playing slots. Given a sack of coins and several slot machines with unknown payout ratios, the bandit must decide how to distribute the coins to get the highest payout. At first, the bandit does not know what each machine will pay out, but as she pulls each one, she gets a better understanding.
See the definition of regret minimization below for a solution to the multi-armed bandit problem.
|
| Multi-page funnel testing | Testing that includes different changes across multiple pages to measure the difference in visitor behavior across a funnel (such as product > cart > checkout) or across a whole site. In multi-page funnel testing, when a visitor sees a variation on one page, that visitor will continue to see the variation across all other pages in the experiment. |
| Online marketing | Strategy for leveraging web-based channels to spread a message about a company’s brand, products, or services to potential customers. |
| Optimization strategy |
The optimization strategy defines the goal or objective a user chooses when enabling Stats Accelerator. We offer two optimization strategies: time minimization and regret minimization. |
| Organic search | When a visitor clicks a non-paid link in search engine results to go to a website. |
| Original (or baseline) | Experience that is used to compare against a variation in an experiment or campaign. |
| Pageview event or pageview goal | Event or goal that measures whether a visitor views a specified page. |
| Pause | Stop running an experiment or campaign. |
| Plan | Optimizely pricing and features package. |
| Preview tool | Optimizely feature that lets you toggle between variations, evaluate code execution timing and navigate your site without re-initiating to make sure your variations look and work the way they should. Check out this article for information about the Optimizely Preview tool. |
| Primary metric | The most important goal of your experiment—the action that you most want visitors to take. The primary metric determines which variation in your experiment is the winner. |
| Project | The subsection of your Optimizely account. Each project has its own snippet, set of experiments, and set of collaborators. Learn more about projects in Optimizely. |
| Project Owner | Optimizely collaborator or user role with access to all features for a specific project, except billing. The Project Owner role is similar to the Administrator role but does not have access to billing. |
| Query parameter | Letters and numbers included in the query string portion of a URL that pass data to web applications. They look like this: ?item=true You can use query parameters to optimize based on paid ad campaigns and search engine marketing. You can also force behaviors with query parameters. |
| Query string | Sring of query parameters, separated with an ampersand. They look like this: ?item1=true&item2=false |
| Rail | Smaller column on the left or right side of a webpage that is not part of the "body content." |
| Redirect experiment |
An experiment where one or more variations redirect the visitor to a different URL from the original page (also sometimes referred to as a "split URL test"). Redirects are often used when testing two entire pages against each other, for example, in a landing page test or site redesign. |
| Regret minimization | A framework or strategy that seeks to minimize the number of times a new visitor is shown a sub-optimal experience with respect to the primary metric. In doing so, the framework minimizes the level of quantified “regret” observed after the experiment's conclusion and maximizes the payout. Regret minimization is the traditional approach to solving a multi-armed bandit problem. |
| Regular Expressions (or RegEx) | The syntax used to match text strings. Regular Expressions (or RegEx) is an Optimizely match type you can use for URL Targeting. |
| Results page | Section of the Optimizely interface where you can see the results of your experiments and campaigns and determine the winning variation. For more details, see this article about the Results page in Optimizely. |
| Revenue per visitor (RPV) | The average dollar amount spent by each customer on a website or mobile app. RPV is a common metric for e-commerce sites. |
| Search engine marketing (SEM) | Paid search campaigns. In SEM, companies pay search engines like Google to show their paid links when certain search conditions are met. |
| Search engine optimization (SEO) | Techniques performed to get a website or webpage to display higher in natural search results. Examples of SEO include incorporating keywords into content and making sure keywords appear high on the page. For information about A/B testing and SEO, check out this article. |
| Secondary metrics | "Next-best" actions that you want visitors to take, after the event that's set as your primary metric. Secondary metrics typically occur "up the funnel" and are leading indicators of success. In Optimizely, the metrics you have set in places 2 through 5 are your secondary metrics. |
| Segmentation or visitor segmentation | Analyzing only a subset of traffic in an experiment's results to see if that subset's behavior differs from the average. On your Results page, you can segment results by audience or dimension, depending on your plan. |
| Session | A session is a period of activity for a user. An existing session ends, and a new session begins after 30 minutes of inactivity on the site. The maximum session length is 24 hours, when a new session automatically begins (even if the user was not inactive for 30 minutes). Learn more about how Optimizely counts conversions. |
| Shopping cart abandonment | When a potential customer starts the check-out process for an online order but drops out of the process before completing a purchase. |
| Simple match | URL match type that will match a given URL, without considering https://, query parameters, and other common URL modifications. A simple match is an Optimizely match type you can use for URL Targeting. |
| Snippet (or Optimizely snippet) | Line of JavaScript code that you place on your page so you can use Optimizely. Learn more about the Optimizely Web Experimentation Snippet. |
| Split URL test | See redirect experiment. |
| Statistical significance | Statistical significance is a measure of how unusual your experiment results would be if there was actually no difference between your variation and baseline and the difference in lift was due to random chance alone. |
| Stats Engine | Optimizely's proprietary way of calculating experiment results that allows you to interpret results without waiting for a fixed test duration and reduces the chance of false positives. |
| Substring match | URL match type that will match a string within a URL. For example, 'ample' is a substring match for 'example.com'. Substring match is an Optimizely match type you can use for URL Targeting. |
| Taguchi method | Reducing the number of variations in a multivariate test (MVT) so that not every single variation has to be tested. Taguchi method MVT results in displaying the "best predicted" variation, even if that variation was not actually tested. Optimizely uses full factorial MVT instead of Taguchi method MVT. |
| Targeting | Determining whether visitors will be included in an experiment. In Optimizely, URL Targeting determines whether an experiment will run on a certain URL, and Audiences determine whether an experiment will run for visitors with certain characteristics. |
| Time minimization |
A framework or strategy that aims to identify and distribute visitors to variations with a relative likelihood of reaching statistical significance that is higher than other variations. To be technically precise, time minimization addresses a set of problems not defined as multi-armed bandit problems.
Optimizely developed this framework, Stats Accelerator.
|
| Unique visitors | Distinct individuals who visit a page during a given period, regardless of how often they visit. Unique visitors are distinct from visits, which are the number of times anyone has visited your site within a given period. A unique visitor may have several visits. |
| URL (or web address) | Abbreviation for Uniform Resource Locator. The web address of a website, as you would see it in your browser's address bar. |
| URL Targeting | Method of specifying the pages where your experiment or campaign will run. |
| User flow | The path taken by a prototypical user on a website or mobile app to complete a task. |
| Value proposition or "value prop" | Essence of your product or service's value to visitors and customers. |
| Variation | Alternate experience that is tested against the original (or baseline) in an A/B test. |
| Viewer | Optimizely collaborator or user role that can only view experiments and campaigns. |
| Visitor | Individual person who goes to your website. |
| Visits | The number of times anyone has gone to your website within a given period. A unique visitor may have several visits. |
| Visual Editor | Section of the Optimizely Editor that allows you to edit your variations visually, using a WYSIWYG (or "what you see is what you get") editor. See this article for more information about the Visual Editor in Optimizely. |
| Winner or winning variation | Best-performing variation of a test based on the most important metrics. An experiment or campaign may have multiple winners if no single clear metric defines which variation will be implemented. Refer to Take action based on the results of an experiment for details about determining the winner or winning variation. |
<%= heading %>