Walkthrough for Bulk Insert

This walkthrough guides you through the steps for loading records using the Bulk API.

Before you can start, make sure you’ve completed the prerequisites required to set up your Ingestion API:

Setup Ingestion API connector
Create and Deploy Data Stream
Configure connected app

Create a CSV file that has a header row matching the fields in the data stream you defined. For example, orders.csv. Each file shouldn’t exceed 150MB.

1id,contact_name,created_date,tax_exempt,ship_address,total,tax_rate,modified_date
2f19cae7e-5cdb-41ce-92ba-539afea04f9d,Jadyn Kozey,2021-07-22T08:57:08.118Z,false,9818 Walter Course Apt. 811,77506,84,2021-07-05T09:31:44.457Z
348c8b6a6-8179-4a15-ba19-ec5c8206ce28,Kristina Jakubowski,2021-07-08T22:57:02.008Z,false,245 Schowalter Port Suite 245,16986,61,2021-07-03T06:20:52.886Z

Request a Customer Data Platform access token. The access_token property in the token exchange response contains the bearer token to use for the authorization header. The instance_url is the Customer Data Platform instance where the API is hosted.

To upload the data, first create a job. In this example, we’re upserting Orders data from our ecomm Ingestion API connector. If you’re deleting records, you would use the delete operation instead. You should get a JSON response that includes the job id, with a job state of Open. You’ll need the job id in the next steps.

Request

1curl --location --request POST 'https://{instance_url}/api/v1/ingest/jobs' \
2--header 'Authorization: Bearer {access_token}' \
3--header 'Content-Type: application/json' \
4--data-raw '{
5   "object":"Orders",
6   "sourceName":"ecomm",
7   "operation":"upsert"
8}'

Response

1{
2   "object": "Order",
3   "id": "913a8b00-7a9e-4331-aaf6-a019ae5021ab",
4   "operation": "upsert",
5   "sourceName": "ecomm_api",
6   "createdById": "005R0000000zqc8IAA",
7   "createdDate": "2021-08-12T17:06:42.921387Z",
8   "systemModstamp": "2021-08-12T17:06:42.921387Z",
9   "state": "Open",
10   "contentType": "CSV",
11   "apiVersion": "v1",
12   "contentUrl": "/api/v1/ingest/jobs/913a8b00-7a9e-4331-aaf6-a019ae5021ab/batches"
13}

Upload your CSV data. After creating a job, you're ready to upload your data. You provide record data using the CSV file you created earlier. You can submit up to 100 CSV files as part of a job.

Request

1curl --location --request PUT 'https://{instance_url}/api/v1/ingest/jobs/913a8b00-7a9e-4331-aaf6-a019ae5021ab/batches
2' \
3--header 'Authorization: Bearer {access_token}' \
4--header 'Content-Type: text/csv' \
5--data-binary '@orders.csv'

Response

1201 Created

Upload complete. After you're done submitting data, you can inform Customer Data Platform that the job is ready for processing by closing the job.

Request

1curl -v --location --request PATCH 'https://{instance_url}/api/v1/ingest/jobs/913a8b00-7a9e-4331-aaf6-a019ae5021ab' \
2--header 'Authorization: Bearer {access_token}' \
3--header 'Content-Type: application/json' \
4--data-raw '{
5"state" : "UploadComplete"
6}'

Response

1{
2   "object": "Order",
3   "id": "913a8b00-7a9e-4331-aaf6-a019ae5021ab",
4   "operation": "upsert",
5   "sourceName": "ecomm_api",
6   "createdById": "005R0000000zqc8IAA",
7   "createdDate": "2021-08-12T17:06:42.921387Z",
8   "systemModstamp": "2021-08-12T17:36:42.921387Z",
9   "state": "UploadComplete",
10   "contentType": "CSV",
11   "apiVersion": "v1",
12   "contentUrl": "/api/v1/ingest/jobs/913a8b00-7a9e-4331-aaf6-a019ae5021ab/batches"
13}

Check the job status and results. To get basic status information on a job, such as the overall job state or the number of records processed, use a Get Job Info request. The job state changes from UploadComplete to either JobComplete or Failed after the files have been loaded into the data lake.

Request

1curl -v --location --request GET 'https://{instance_url}/api/v1/ingest/jobs/913a8b00-7a9e-4331-aaf6-a019ae5021ab' \
2--header 'Authorization: Bearer {access_token}' \

Response

1{
2   "object": "Order",
3   "id": "913a8b00-7a9e-4331-aaf6-a019ae5021ab",
4   "operation": "upsert",
5   "sourceName": "GreasyConnector",
6   "createdById": "005R0000000zqc8IAA",
7   "createdDate": "2021-08-12T17:06:42.921387Z",
8   "systemModstamp": "2021-08-12T18:06:42.921387Z",
9   "state": "JobComplete",
10   "contentType": "CSV",
11   "apiVersion": "v1",
12   "contentUrl": "/api/v1/ingest/jobs/913a8b00-7a9e-4331-aaf6-a019ae5021ab/batches",
13   "retries": 0,
14   "totalProcessingTime": 0
15}

Customer Data Platform Developer Guide

Walkthrough for Bulk Insert