Upsert Large Volumes of Data Asynchronously

If you plan to perform large upsert operations on data extensions, we recommend that you use batched, asynchronous requests in SOAP API. A batch request contains multiple rows of data in a single request. In a batch request, all of the rows in the API call are processed, even if some rows result in errors.

Configure the API call so that it only sends an HTTP postback when an error occurs. To improve efficiency, configure postbacks to omit error details and the object details from the original request. These settings create an optimal payload size for postbacks. Optimizing the payload size ensures that your servers don’t become overloaded with postback data. The API server doesn’t retry postbacks that fail, so optimizing the size of postbacks reduces the chances of missing an error notification.

To optimize HTTP postbacks, use these settings in the AsyncResponse object in the Options object.

  • Set the value of the ResponseType property to HTTPPost. This setting ensures that when the system finishes processing the request, it notifies the client by sending an HTTP POST call to it.
  • Set the value of the ResponseAddress property to your postback URL.
  • Set the value of the RespondWhen property to OnError. This setting tells the system to only send a postback if an error occurs.
  • Set the value of the IncludeResults property to false. This setting prevents the inclusion of the objects from the request, which significantly reduces the size of the notification.

This SOAP envelope shows an example of a CreateRequest call that’s configured to optimize the size and frequency of HTTP postbacks.

If your upsert call results in an error, issue a RetrieveRequest call that includes the request ID. The response lists the items in the batch that failed. You can then retry the objects that failed, rather than upserting the entire batch again. This process results in better performance and requires you to make fewer API calls.

When a failure occurs, the body of the HTTP postback contains a RequestID value. Use this ID in the query in your Retrieve request.

You can retrieve detailed results by filtering for the request ID and an error code that’s greater than 0 in the FilterPart object in the RetrieveRequest object. The API returns a ResultItem object, which contains an OrdinalID value for the object that resulted in failure. Use the ordinal ID to identify the row that failed.

This example SOAP envelope contains a Filter element with two operands. The first operand ensures that only results for the specific request ID are returned. The second operand ensures that only results where the error code is greater than 0 are returned. An error code greater than 0 indicates that an error occurred.

Postbacks are encrypted in transit using HTTPS. For additional security, configure the URL in the ResponseAddress property to include query string parameters, and use those parameters for client-side processing. The API server isn’t able to retrieve credentials from another source before it sends the postback. However, you can provide the credential values in the query string parameters, as shown in this example.

Another way to improve security is to use IP allow-listing. Configure your server to only allow postbacks from Marketing Cloud Engagement.