Optimizely Feature Experimentation Release Notes

The following is a cumulative list of enhancements and major bug fixes in each Optimizely Feature Experimentation release.

Because Optimizely Feature Experimentation releases new features and fixes as soon as they are ready, this list will update regularly. Follow this article to receive notifications as soon as new content is added.

March 2023

Enhancements

• Improved the layout of the Flags list for better usability.
• Rules are set as paused on creation, which enables staging rules at scale.

Bug Fixes

• Fixed an issue where the Flags UI dashboard was not loading for some customers.
• Fixed an issue where a 500 error was being reported when trying to change the reporting date range in Program Management.

November 2022

New Feature Experimentation SDK

Released an SDK for Flutter 

• Flutter SDK quickstart guide
• Flutter SDK reference documentation
• Flutter SDK is available on GitHub

October 2022

Optimizely Feature Experimentation Parity Items

The following have been released to Optimizely Feature Experimentation, increasing the feature parity between the new Optimizely Feature Experimentation and legacy Optimizely Full Stack Experimentation products. 

• API Values (open beta)
• Project Management Linking (general availability, gradual rollout)

August 2022

Optimizely Feature Experimentation Parity Items

The following have been released to the Optimizely Feature Experimentation, increasing the feature parity between Feature Experimentation and the legacy version of Full Stack. 

• Mutual Exclusion
• Multi-armed Bandit Optimizations
• Stats Accelerator

For more information on the Optimizely Feature Experimentation and Full Stack legacy feature parity work, refer to the developer documentation.

New Features

Created templates and starter kits to implement feature flagging and experimentation across major serverless edge compute platforms.

• • Akamai EdgeWorkers
  • AWS Lambda@Edge
  • Cloudflare Workers
  • Fastly Compute@Edge
  • Vercel Edge Functions

July 2022

New Features

• Created a Helm Chart for running Optimizely Agent on Kubernetes. 

Outlier Smoothing

To continue making our product better and easier to use for everyone, we have made a slight UI update to communicate better how Optimizely's Stats Engine handles outliers in your revenue metrics. There have not been any underlying changes in our Stats Engine or how Optimizely calculates results. This update is purely a phrasing enhancement to improve our user's understanding of what happens to their outlier values.  

What is an outlier?  

An outlier is an abnormally lower or higher value than other values in your results. 

What are Revenue Metrics?  

In Optimizely, a metric is a quantitative measurement of a visitor's action. Metrics are created out of events, which directly track actions like clicks, page views, form submissions, purchases, and scroll depth.  

The total revenue metric tracks the total revenue generated from user interactions with your event. Revenue usually is not your primary metric because, unlike clicks or page views, revenue does not measure a specific, discrete action taken by your visitors. However, tracking revenue is a fantastic way to tie your optimization efforts to the metrics that your company values most. 

What is changing? 

We are only updating the Optimizely UI and documentation to read "Outlier Smoothing" instead of "Outlier Filtering." We believe this update to the phrasing will help users understand what is happening with their outlier values.  

Why smooth outliers in revenue metrics?  

Outliers can severely skew the accuracy of any analysis conducted on a data set and can lead to potentially incorrect conclusions. Smoothing outliers results in more accurate findings. We recommend turning on this setting for your revenue metric to improve the integrity of your results. 

How does Optimizely treat outliers? 

For revenue metrics, Optimizely gives you the option to use outlier smoothing. Optimizely's outlier smoothing algorithm first identifies any values exceeding the daily exclusion threshold, extreme values three standard deviations higher than the observed mean. These extreme values are designated as outliers. Next, Optimizely replaces these outliers with the metric's harmonic mean value. This step in the process is known as outlier smoothing. Optimizely recalculates the daily exclusion threshold for each day using a moving average of your metric's arithmetic mean and standard deviation over the previous seven (7) days. This process repeats each day of the experiment. Please view our documentation on how Optimizely handles outliers for more detailed information.  

Conclusion 

If you have any questions or feedback, feel free to email us at support@optimizely.com! Keep on optimizing!

Forced Decision Methods

Here at Optimizely, we have been working hard to start 2022 with significant upgrades to our QA experience and security. We are proud to announce the release of new API methods for our Feature Experimentation SDKs and some critical security upgrades for Python. Thank you for reading this month’s release notes, and we are excited to see all the new places and faces 2022 will bring us!

Forced Decision Methods

We’ve recently rolled out a set of new APIs for our Feature Experimentation SDKs that will make overriding and managing user-level flags, experiments, and delivery rules even more straightforward.

These new methods extend our OptimizelyUserContext object, which previously allowed you to make flag decisions and flag events for a specific user. Now we’ve taken things up a notch!

The new methods allow you to do the following:

• setForcedDecision—Forces a user into a specific variation
• getForcedDecision—Returns the variation the user is forced into
• removeForcedDecision—Removes a user from a particular variation
• removeAllForcedDecisions—Removes a user from all forced variations

Using these “Forced Decision” methods makes it even easier to set up automated testing and QA by forcing certain User IDs into specific variations regardless of audience conditions and previously configured traffic allocations.

For more detailed information, please click on the SDK you are interested in to view the developer documentation on the Forced Decision methods:

• Android SDK
• C# SDK
• Go SDK
• Java SDK
• JavaScript (Browser) SDK
• JavaScript (Node) SDK
• PHP SDK
• React SDK
• React Native SDK
• Ruby SDK
• Python SDK
• Swift SDK

Vulnerability and Python Support Upgrades

As part of Optimizely’s regular security updates and modernization of the Optimizely Feature Experimentation product, we will no longer officially support older versions of Python. These older versions do not provide secure libraries needed for the Optimizely Python SDK. For example, Python version 3.4 support has ended due to a known security vulnerability in the PyYAML library.

Optimizely supports the following versions:

• Python 3.7
• Python 3.8
• Python 3.9
• Python 3.10 and above
• PyPy3

Developer Documentation UI Updates

Last month’s release notes announced that we are working on exciting UI changes to our developer and end-user documentation. If you visited our docs recently, you might have noticed some updates, including a completely new UI redesign. We’re happy to report that these changes are going well and will be fully released soon.

Conclusion

If you have any questions or feedback, feel free to email us at support@optimizely.com! Keep on optimizing!

Allowlisting

Quality assurance just got easier for Optimizely Feature Experimentation customers. Optimizely is excited to announce that Allowlisting is now available to all customers. Allowlisting, previously known as whitelisting, was available in the previous version of Full Stack, so we are excited to continue putting our developers first by fully releasing Allowlisting in our Flags experience.