Test Commerce Search v3

  • Updated

This feature is currently in beta. Contact your Customer Success Manager or sign up for the beta on Optimizely.com.

After you implement Commerce Search v3 for Optimizely Configured Commerce, you can test the default controls to see if you need to edit any settings.

To ensure that user events effectively influence search results, the Google Cloud Vertex AI Search must reach a certain threshold of total user events ingested into its Retail Search project. This threshold is necessary for the AI to be adequately trained on the events and start influencing the search results, thereby providing more relevant results. During the initial phase, the AI may not immediately reflect the impact of user events on search results until sufficient data is collected and processed.

Monitor user events flow

Ensure that the Daily User Events Sync integration job runs successfully. This guarantees that user events sent from the Commerce storefront and backend to Optimizely Data Platform (ODP) are synced to Google Cloud Vertex AI Search (Retail Search).

Rebuild index

  1. Go to Marketing > Indexing in the Admin Console and click Rebuild All.
  2. Check the Indexing Jobs log to monitor the progress of product catalog ingestion into Retail Search. Review the logs for any error messages related to products that failed to index and examine all error and warning messages.
  3. Update the product details to correct any errors, then rerun the Rebuild Index job.
  4. Ensure the process completes successfully. Report any unresolved indexing errors or warnings to Optimizely Support.
  5. (For multiple languages and websites) Verify that the job successfully indexed each website's product catalog to Retail Search for all active website languages. 

You should rebuild a couple of times to sync product attribute configuration and searchability.

Monitor how user events influence search results

Over time, verify that user events (such as searching or browsing products, adding products to the cart, purchases, viewing product details, and so on) influence the Google Vertex AI search engine to return more relevant search results.

  1. Log in to the storefront and assign a customer ship-to and bill-to. Use this customer context throughout these steps. The user can be anonymous when logged out, but all events triggered are associated with the login afterward.
  2. Perform searches on the storefront as defined in Test search.
  3. Perform the following actions on the storefront that would track and send user events:
    • Add products to cart.
    • View product details.
    • View the cart.
    • Search products and browse categories.
    • Sort products on the Product List page.
    • Filter products on the Product List page.
    • Select a search suggestion from the autocomplete results in the search bar.
  4. Verify that products are ranked higher in search results based on past clicks.
  5. Verify that products with higher purchase rates are ranked higher in search results.
  6. Verify that products with higher user interactions (such as adding to cart and viewing product details) are ranked higher in search results.
  7. Verify that recent user events influence search results.
  8. Verify that long-term user events have a lasting influence on search results.
  9. Verify that products with negative user events (such as no detail view and not adding to cart) are ranked lower in search results.

Test text search and browse search

Use the following list to help test your searches and browses after successfully rebuilding the index:

  • Browse through the category menu and view products from different categories. Review how the products are listed by their relevancy on the Product List page. Evaluate how Retail Search AI intelligently orders the products and provide any useful feedback on this relevance.
  • Go to the Brands pages and review how the products are listed by their relevance.
  • Search using partial terms, words, prefixes, phrases, full keywords/terms, and natural language that may potentially match any of the following product fields:
    • Title
    • Description
    • Part Numbers (like ERP, Manufacturer Item Number, and Model Numbers)
    • Attributes (like color and size)
    • Any other searchable product fields
  • Select and deselect filters on the Product List page and review how the products are listed and sorted by the best relevancy determined by Retail Search AI.
  • Sort products using different available sort order options and review how the products are sorted. Review section Semantic Embedded Based Filtering affecting search results when sorting by default and non-default sort option.
  • Apply different pagination size options and go through the subsequent pages.
  • Test how the Retail Search AI suggests spelling corrections when searching with terms that may be misspelled or when a correct term might potentially match products in the catalog. Test how Configured Commerce does not suggest corrections to search terms that are part numbers when Exclude part number from autocorrect is turned On.
  • Test searches with the Search Query Expansion setting turned off or on. Perform the previous steps and review how the products are listed by relevancy by Retail Search AI.
    • Query expansion increases recall for query terms with few results, especially long-tail queries. For example, searching Google Pixel 5 without query expansion only returns google_pixel_5. With query expansion, you may also get google_pixel_4a_with_5g, google_pixel_4a, and google_pixel_5_case.
    • When a shopper uses an ambiguous or multi-word search phrase, they may get an empty response. With query expansion turned on, the request is analyzed, and the product list is expanded based on the search query.

Search on a specific website or language

  • Website-specific catalog indexing
    • Each website has its catalog indexed to its associated Retail Search projects by website and language.
    • The search results should only include products from the specific website being searched.
  • Language-specific search
    • Ensure that the AI understands the context of language-specific searches.
    • Ensure that the search functionality is consistent across different languages.

Autocomplete

Go to the storefront and test the autocomplete. Enter different search terms in the search bar and wait for any autocomplete suggestions to display.

The following results are expected to display based on the search term. You may not see most suggestions until the autocomplete model trains on events.

  • Search - Suggestions
  • Categories - Popular
  • Categories - Suggestions
  • Brands - Popular
  • Brands - Suggestions

Autocomplete triggers after entering two characters. Test the following scenarios:

  • Common search terms (like shoes or laptop)
  • Synonyms (like sofa and couch)
  • Related terms (like phone and charger)
  • Brands (like nike and reebok)
  • Categories (like grinders and faucets).

You should also verify suggestions for different languages on various websites by entering search terms in different languages (like English, Spanish, and French).

Dynamic facets

Dynamic facets are search filters, specifically attribute type filters, that are automatically generated and ranked by the search system based on user interactions and the relevance of the search results. These facets are dynamically generated by AI to help users refine their search results more effectively.

Attribute type filters are the only ones dynamically generated by Retail Search. Filters such as Categories, Brands, Stocked Items, Price, and Product Lines are all system filters and are not dynamically generated.

Observe the dynamic facets that may display in the following conditions:

  • When searching with different search terms.
  • When browsing by category.
  • When browsing by brand.
  • When filtering or sorting product results.

Semantic embedded-based filtering

With default sorting, Best Match, Retail Search displays a range of search results, including products that are popular or trending, even if they are slightly relevant. With non-default sorting, like Product: A to Z, some products may display higher in the results even if they are not the most relevant to the search query. To improve the quality of non-default search results, Retail Search uses semantic embedding-based filtering. This technique helps to filter out less relevant items, ensuring that the search results are more relevant to the user's query. While this filtering improves the relevance of search results, it may also reduce the total number of search results.

There may be instances where sorting by a non-default sort option, such as Product: A to Z, results in no products being displayed. This is expected behavior.