Optimizely's results page does not sample from your data

  • Updated
  • Optimizely Web Experimentation
  • Optimizely Performance Edge
  • Optimizely Feature Experimentation
  • Optimizely Full Stack (Legacy)

Sampling involves choosing a subset of data from a complete dataset to explore and uncover untapped insights regarding the broader population. Optimizely Experimentation's results page does not sample from your data. The results page includes ALL of your experiment data.

No events are excluded at Optimizely

Optimizely's Results Page API reads all recorded events in an experiment into the results page.

There is nothing in Optimizely's Results API that excludes or takes a portion of the events.

The sampling split among variations is random but not simple

The user randomization procedure uses bucketing to assign different users to different variations. This process depends on a user ID and attributes.

Optimizely uses a unique value for each experiment to bucket each visitor with a MurmurHash function which makes a hash of the user ID and experiment ID.  As a part of the algorithm, Optimizely assigns each user a number between 0 and 10,000 to determine which bucket they are assigned and, subsequently, what variation that user sees. The large hashing range allows assigning traffic granularity at fine increments of 0.01%.

Because MurmurHash is deterministic, a user ID always maps to the same variation as long as the experiment conditions do not radically change. A given visitor has a consistent bucketing ID within the campaign. The ID persists across channels and is unaffected by poor network connectivity. However, a visitor does not have the same bucketing ID across campaigns. See Ensure consistent visitor bucketing for more information.

For Optimizely Feature Experimentation, you can implement a User Profile Service to persist bucketing IDs across SDKs.

If you filter a date range, you see your actual data

When you filter your data with the date range filter, you see the true data truncated within the specified range. Optimizely does not perform any sampling or extrapolation on data.

Learn more about how bucketing works at Optimizely.