In 2019, we announced the Import API – Pardot’s first asynchronous API. We’re happy to announce a follow up in the Export API. Similar to the Import API, the Export API is designed to help developers work around standard bulk data limits. Using the conventional API, users can query objects and receive 200 records per request. By making use of the new Export API, users can retrieve data for millions of records with a handful of API calls. Starting in Spring ‘20, we will release export endpoints for Visitor Activity. In this blog, we walk you through the usage of the new endpoints, provide code samples, and show you how to make the most of your API limits.
Terminology
The Export API uses the term “export” as a unit of work for all records handled in one request, initiated by the user. Each export will be split into several smaller queries for ease of processing. Results for each query are stored as CSV files. In Pardot, exports and their associated files have a one-to-many relationship. When reading a given export object, an array of references to the files will be returned.
Exports have a notion of expiration. After an export is complete, the user who initiated the export will have 7 days to download the results. After which, the export record will expire and the files will be deleted. Users can still read the export record via the API, however no results will be returned. In addition, any export that does not complete its processing after 7 days will be canceled and the results expired.
Object Support
As of this publication, the Export API only supports Visitor Activity. We chose Visitor Activity for our first implementation because it is one of the largest data sets in Pardot, making it a good target for a bulk API. Most activity associated with anonymous visitors and identified prospects is stored here. If the user wants a total history of engagement for their account over a given time span, then exporting visitor activity is a good place to start.
Basic Usage – Examples
First, log in to the Pardot API using your email, password and user_key to get your api_key. To find your user key, log in to Pardot through a browser and navigate to ⚙︎ > Settings > My Profile. Copy the value next to API User Key.
We’ll be using Curl to access the API, which is a simple command line tool to execute HTTP requests and see responses. Most distributions of macOS and Linux have Curl preinstalled. However, on Windows, the installer can be downloaded from the Curl website.
Executing this Curl command returns a JSON response that contains the api_key for the current session. This api_key is used in the Authorization headers when calling the Pardot API.
Sample output from the previous Curl command
Now that we’ve obtained an api_key, we can make calls for one hour before it expires. If writing an integration, it’s wise to cache this value until it expires, rather than retrieving a fresh api_key for every call.
Let’s create an export. We’ll call the create endpoint and specify a time range. Note that there is a limit of 1 year of data that can be pulled with a single export. There is no limit on the number of rows in a given export.
In order for the system to know which object we want and the desired time range, we’ll specify an Object and a Procedure. Check the official documentation for the latest set of supported objects and procedures. For now, we’ll specify VisitorActivity and filter_by_created_at. Each export endpoint mandates an object and a procedure in order to filter the data and provide fast retrieval. In general, a procedure corresponds to a field on the object. The procedure’s arguments, supplied by the user, tell it how to filter the data.
Note that the dates specified do not have to fall exactly on the new year. This example was chosen for convenience.
Once the create request is sent and successfully acknowledged, a new export record is created and the export is put into the queue for processing. Note that for a given account, only one export at a time will be processed. Exports will be processed in the order they were created.
Upon successful acknowledgement, a response is returned (below). The response will bear the contents of the export object, including the ID, state, and original arguments that created the export. This information can be retrieved at any time by accessing the read endpoint, along with the ID. Keep a reference to the ID value, as it will be required later. Here’s an example of a successful response:
You can think of the export processing system as a state machine, where exports only flow in one direction. When the export is created, it will be in a status of Waiting. Once the export is picked up from the queue for processing, it will move into the Processing state. After processing, it will move to Complete.
Users can check the state of the export by polling the read endpoint:
Keep in mind that you’ll have to substitute the {id} and {version} tags with their appropriate values.
Note that polling the read endpoint will cost API calls, so be sure to budget your calls in accordance with your account limits.
Once complete, the response will look like:
Note that resultRefs may not be an array if there is only one file.
Once the export is complete, you can iterate through the resultRefs and call the corresponding endpoints, like so:
To keep your data safe, Pardot acts as an authorization gateway for the files. The files are also encrypted. If a file’s URL were to be exposed, it would not be useful without the decryption key, which Pardot stores for a limited time and then discards. The files themselves are also deleted after the export expires.
The output of this endpoint will be a CSV with a header row. Here’s an example:
Please note that the ordering of the results is not guaranteed. In other words, if there are two IDs in the result set, the lesser ID may come after the greater ID.
For larger datasets, result CSVs can be several megabytes in size. Download times will depend on your network speeds.
Performance & Limitations
To achieve its performance, the Export API makes tradeoffs in other areas. Typical API flexibilities and conveniences have been eliminated in favor of speed. Users of the Pardot API may be familiar with the fact that when reading Visitor Activities, data from other objects is joined into the response in order to make reading the response more convenient. This is not the case with the Export API. References to data in other tables will be included as IDs only. Fetching the data from other objects will require additional API calls. This helps keep the size of the result files as small as possible, while minimizing expensive queries to fetch additional data for each row. If you want those additional details, you can use the appropriate endpoint to retrieve them.
The Export API is designed to deliver a large, complete result set in an asynchronous fashion. While it can handle processing millions of records at a time, it is not designed for real time speed. If exporting millions of records, you can expect the results to take over an hour to populate. If you are looking to capture near-real-time changes to visitor activity, you may want to try pulling smaller result sets via the synchronous API instead.
What’s Next
Get those keyboards ready – you can start using the Export API today! We’re glad to provide our developer community with more ways to get data in and out of their Pardot accounts. We look forward to expanding upon the Export API in future releases. Check out the official documentation and release notes to see what else is new in Spring ’20.