Closed-loop interface

  • Updated

This topic describes the closed-loop interface that is used for bi-directional handling of actions and data exchange between Optimizely Campaign and external systems. For example, using the closed-loop interface, you can create and send personalized newsletters automatically in Optimizely Campaign from an external campaign management system.

Setup

Contact customer support regarding setup of the closed-loop interface, and terms and conditions that apply when accessing this type of non-anonymized action-based data.

Because the response data consists of non-anonymized action-based data, you have to indemnify and hold Optimizely harmless in advance from any potential liabilities and third-party claims which may result from making this functionality available to you.

How it works

The closed-loop interface consists of two modules: the import module and the response data export.

To create and send mailings automatically, you must first create a source mailing in Optimizely Campaign and submit the mailing ID to your external system, for example, a campaign management system (CMS). In the CMS, you select the recipients and provide the recipient data.

The import module transfers the recipient data including the ID of the source mailing to Optimizely Campaign. After the import, you receive a status notification. In Optimizely Campaign, a campaign copy is created with the personalized content (for example, product recommendations) and the mailing is sent - either directly, or at a specified time. Optionally, you can resend the mailing manually if recipients have generated soft bounces, for example.

The response data export transfers the response data (opens, clicks, responses, unsubscribes) of the sent mailing back to your CMS. You can then process the response data further in the CMS or a third-party system, such as a data warehouse.

Image: Closed loop diagram

The import module and the response data export can be configured and used independently of each other.

Exchanging data

The data exchange is done using an SFTP server set up at Optimizely, and recipient data is transferred to that server. Response data is available for download from the same server.

Running transfers must have a temporary file name to avoid incomplete file exchange, see Data transfer troubleshooting. Naming conventions are described below.

The data exchange does not respect strict separation of clients. All user data can be transferred through one SFTP account. You can set up client-specific SFTP accounts if required for security reasons. Alternatively, you can set up separate sub-directories in one client. In both cases, the recipient data import module must then commit one file per client/directory.

Transferring recipient data

Recipient data can be transferred at any time to the Optimizely Campaign SFTP server.

The name convention of the transferred files must be as follows:

YYYYMMDDHHMMSS_subscribers.csv

The recipient file is a CSV file with the following properties:

  • Phrase and field lengths are variable within the limits of the recipient list.
  • A semicolon (;) is used as separator.
  • Line feed is used for line breaks.
  • Quotation marks ("") are used as field boundaries. To designate quotation marks within a field, wrap them in quotation marks.
  • The file does not contain declarations regarding columns and field lengths.
  • Empty fields (NULL) remain empty in a data set. The number of field separators must be constant.

Recipient data is processed automatically every 10 minutes. The SFTP server of Optimizely Campaign provides log files for download within 30 days after creation.

According to the specifications, fields of the main recipient list must be committed. Fields supposed to not contain any value are committed as empty columns. Existing fields and the field sequence are fixed. Do not change them without the consent of both parties.

Using personalization and recommendations

It is possible to enhance the delivered recipient data with personalization, for example by adding product recommendations to the mailing.

One or more fields reserved for a product ID are added to the CSV file with the recipient data. Product information (such as title, description, image-URL) for recommended products is transferred in a second CSV file. Product information is inserted into the mailing using field functions and a placeholder. Assigning product IDs to recipients is done by the customer.

With web analysis software, recommendations are available via a CSV file. Other recommended products are assigned to each product. Product information is rendered into the mailing using a special field function, similar to the process described above. You do not need to assign products to a recipient, as this is automatically done by the web analysis software in accordance with the initial configuration.

To add personalized content, you need a new template. Contact customer support.

Creating source mailings and sending mailings

As a basis for the campaign you are going to send, you create a source mailing in Smart Campaigns and submit the mailing ID to the external system. A so called master list serves as recipient list for the source mailing. Placeholders referencing a specific field of the recipient list are inserted into the mailing paragraphs. Using these placeholders you can individually assign and insert product data and other information into the mailing.

To send multiple newsletters, or send newsletters at the same time, you have to create multiple source mailings. The source mailing is never sent and must remain in status New or respectively Activation required. Do not activate the mailing, otherwise the copy cannot be created. Instead the system automatically creates a copy of it and sends that, so the source mailing can be reused.

If you create a mailing using the REST API, you cannot use the mailing for the closed-loop interface. Create the source mailing only in Smart Campaigns.

Triggering the dispatch

When a file containing recipient data is stored in the corresponding directory on the SFTP server, the mailing is sent. Every 10 minutes, a Cron job checks this directory for new files. If a new file exists, it is imported into the recipient list of the corresponding client, and assigned to the source mailing using the parameter BROADMAIL_ID.

There are different ways to add content to a mailing:

  • Directly from the file. The content is imported along with the recipient data in a CSV file. This method is only applicable for texts and hyperlinks.
  • Referenced content. URLs in the imported CSV files are used to reference the content. When sending the mailing, the content is retrieved from the external server of the customer through the content interface. You can add texts, pictures, attachments, and hyperlinks types of content to a mailing.

When the data is imported, the mailing is automatically sent, and gets a unique ID.

You can send test emails using the closed-loop interface by storing test data instead of real data on the server. The CSV file must have the same structure as the file containing the real recipient data. Every time a test email is sent, a new mailing is created in Optimizely Campaign.

Mailings are triggered only by storing a file on the server configured for the closed-loop interface. You cannot send mailings by using the Optimizely Campaign user interface.

You can customize the source mailing for different newsletters. Only change the source mailing after the previous mailing is sent. Additionally, ensure that the correct mailing ID of the source mailing is used in the recipient CSV file.

Troubleshooting data transfer

If a problem occurs transferring the data, a notification is sent to an email address configured beforehand. An error file is generated and the transfer stopped. During implementation you can configure the exact interface behavior in case of an error. When the implementation is completed, you can switch off this setting.

If an error occurs, the interface must be restarted in accordance with the error file. These are the cases:

  • Data import failed.
  • Error when mailing is started after data import.

Errors during data transfer are handled by customer support, since these can only be resolved manually.

Monitoring and sending status

After a recipient data import, the status of the process is sent via email and registered in a log file. This assures that actions within the interface are properly logged and documented in Optimizely Campaign.

Structure of the main recipient list

Column name Data type Example Description
Email Varchar(255) user@example.com Email address of the recipient
broadmail_ID Bigint/Long 44018617811 Mailing ID in Optimizely Campaign (broadmail_ID)
WAVE_ID Bigint/Long 46623317811 Identifies the selection/sending wave for unambiguous assignment of user actions. This parameter is optionally set by the external system when importing the recipient data.
Salutation Varchar(255)   Field from existing recipient list.
First name Varchar(255)   Field from existing recipient list.
Last name Varchar(255)   Field from existing recipient list.

The list configuration described above is an example. More fields can be added if required for personalization purposes. Contact customer support beforehand, and document them in the interface setup.

Specifying response data

Response data is generated daily from the log files, and is available for download at a configured time via an SFTP server within Optimizely Campaign. The files contain the most recent data. The files containing the mailing data are available for download for 30 days after starting the mailing.

The response data consists of eight CSV files. Each of these can be read as a table. The response data is exported as String data type. You can also convert numeric values such as the mailing and client ID to the Long data type. See detailed type information for each file below.

The names used in the following documentation are examples. File names can be configured.

Sent mailings

YYYYMMDD_mailings.csv contains mailings sent in the referenced period.

Column name Description Example Key
id (mailingId) Mailing ID 51106527229 Primary key
compoundMasterId Campaign ID in Smart Campaigns 71206365463  
mailingGroupId Client ID (BROADMAIL_ID) 44018617811  
mailing.recipientLists Recipient list ID 58243723257  
started Start time of the mailing 2021-01-01 12:07:55  
finished End time of the mailing 2021-01-01 12:09:10  

Sending Log

YYYYMMDD_mailingrecipients.csv contains recipient-related data. The sending log refers to the mailings contained in the file YYYYMMDD_mailings.csv.

Column name Description Example Key
id (mailingToUserId)

Encoded assignment of the recipient

25951836752 Primary key
mailingId Mailing ID 51106527229 Foreign key
mailingGroupId Client ID (BROADMAIL_ID) 44018617811  
userId Email address of the recipient user@example.com  
created Sending date and time 2021-01-01 12:07:55  
sendingStatus Sending status
  • sending - Mailing is sent
  • sent - Mailing was sent
  • could_not_be_sent - Mailing could not be sent
 
deliveryStatus Delivery status
  • OK - Mailing was delivered
  • HARD - Hard bounce
  • SOFT - Soft bounce
 
response.category Response category
  • unknown - The recipient sent a response to the mailing
  • bounce - Soft bounce
  • temporary_bounce - Soft bounce
  • fatal_bounce - Hard bounce
  • autoresponder - Autoresponder
 

Links

YYYYMMDD_links.csv contains links from the mailing. Data is only available for links with activated link tracking.

Column name Description Example Key
id (linkId) Link ID 44018617936 Primary key
mailingId Mailing ID 51106527229 Foreign key
mailingToUserId

Encoded assignment of the recipient

25951836752 Foreign key
mailingGroupId Client ID (BROADMAIL_ID) 44018617811  
name Link name Impressum  
url Link target (URL) http://www.example.com  

Clicks

YYYYMMDD_clicks.csv contains mailing clicks generated by recipients, within the referenced period. Clicked links/URLs can be found in the file YYYYMMDD_links.csv.

Column name Description Example Key
linkId Link ID 44018617936 Foreign key
mailingId Mailing ID 51106527229 Foreign key
mailingToUserId

Encoded assignment of the recipient

25951836752 Foreign key
mailingGroupId Client ID (BROADMAIL_ID) 44018617811  
userId Email address of the recipient user@example.com  
created

Date and time of the click

There can be multiple clicks from one recipient within one second.

2021-01-01 12:07:55  
click.browser Web browser Firefox 64.1  
click.platform Operating system Windows 10  
click.device Device type Desktop  

Opens

YYYYMMDD_opens.csv contains mailings opened by recipients in the referenced period.

Column name Description Example Key
linkId Link ID 44018617936 Foreign key
mailingId Mailing ID 51106527229 Foreign key
mailingToUserId

Encoded assignment of the recipient

25951836752 Foreign key
mailingGroupId Client ID (BROADMAIL_ID) 44018617811  
userId Email address of the recipient user@example.com  
created

Date and time when the recipient opened the mailing

One recipient can open a mailing several times. In this case, each opening generates a singular data set.

2021-01-01 12:07:55  
open.browser Web browser Firefox 64.1  
open.platform Operating system Windows 10  
open.device Device type Desktop  

Automatic opens

YYYYMMDD_automated_opens.csv contains automatic opens in the referenced period. See Apple Mail Privacy Protection.

Column name Description Example Key
mailingId Mailing ID 51106527229 Foreign key
mailingToUserId

Encoded assignment of the recipient

25951836752 Foreign key
mailingGroupId Client ID (BROADMAIL_ID) 44018617811  
userId Email address of the recipient user@example.com  
created

Date and time of the generated open

2021-01-01 12:07:55  
open.browser Web browser Apple Mail Privacy Protection  
open.platform Operating system unknown  

Unsubscribes

YYYYMMDD_unsubscribes.csv contains unsubscribes by recipients in the referenced period.

Column name Description Example Key
mailingId Mailing ID 51106527229 Foreign key
mailingToUserId

Encoded assignment of the recipient

25951836752 Foreign key
mailingGroupId Client ID (BROADMAIL_ID) 44018617811  
userId Email address of the recipient user@example.com  
created Date and time of the unsubscribe 2021-01-01 12:07:55  

Responses

YYYYMMDD_responses.csv contains emails returned to the sender address from any user within the referenced period.

Column name Description Example Key
mailingId Mailing ID 51106527229 Foreign key
mailingToUserId

Encoded assignment of the recipient

25951836752 Foreign key
mailingGroupId Client ID (BROADMAIL_ID) 44018617811  
userId Email address of the recipient user@example.com  
created Sending date and time 2021-01-01 12:07:55  
category Response category
  • unknown - The recipient sent a response to the mailing
  • bounce - Soft bounce
  • temporary_bounce - Soft bounce
  • fatal_bounce - Hard bounce
  • autoresponder - Autoresponder