Bulk Ingestion Walkthrough

This walkthrough guides you through the steps for loading records using bulk ingestion.

Before you can start, make sure you’ve completed the prerequisites required to set up your Ingestion API:

• Set up an Ingestion API connector to define the endpoints and payload to ingest data.
• Download the object endpoints from the configured Ingestion API connector. The object endpoints and the Ingestion API connector name are used as parameters in the API calls.
• Create an Ingestion API Data Stream to configure ingestion jobs and expose the API for external consumption.
• Refer to the links in see also sections to complete authentication steps.

1. Create a CSV file that has a header row matching the fields in the data stream you created. Each file must not exceed 150 MB. Here's some sample data of an orders.csv file used in this walkthrough.

   id,contact_name,created_date,tax_exempt,ship_address,total,tax_rate,modified_date
   f19cae7e-5cdb-41ce-92ba-539afea04f9d,Jadyn Kozey,2021-07-22T08:57:08.118Z,false,9818 Walter Course Apt. 811,77506,84,2021-07-05T09:31:44.457Z
   48c8b6a6-8179-4a15-ba19-ec5c8206ce28,Kristina Jakubowski,2021-07-08T22:57:02.008Z,false,245 Schowalter Port Suite 245,16986,61,2021-07-03T06:20:52.886Z

2. Request a Data Cloud access token. For more information, refer to Authentication. The access_token property in the token exchange response contains the bearer token to use for the authorization header. The instance_url is the Data Cloud instance where the Ingestion API is hosted.

3. To upload the data, first create a job. In this example, we’re upserting orders data from an ecomm Ingestion API connector. If you’re deleting records, use the delete operation instead. You must get a JSON response that includes the job id, with an Open job state. You’ll need the job id in the next steps.

   Request

   curl --location --request POST 'https://{instance_url}/api/v1/ingest/jobs' \
     --header 'Authorization: Bearer {access_token}' \
     --header 'Content-Type: application/json' \
     --data-raw '{
   "object":"Orders",
   "sourceName":"ecomm_api",
   "operation":"upsert"
   }'

   Response

   {
     "object": "Orders",
     "id": "913a8b00-7a9e-4331-aaf6-a019ae5021ab",
     "operation": "upsert",
     "sourceName": "ecomm_api",
     "createdById": "005R0000000zqc8IAA",
     "createdDate": "2021-08-12T17:06:42.921387Z",
     "systemModstamp": "2021-08-12T17:06:42.921387Z",
     "state": "Open",
     "contentType": "CSV",
     "apiVersion": "v1",
     "contentUrl": "/api/v1/ingest/jobs/913a8b00-7a9e-4331-aaf6-a019ae5021ab/batches"
   }

4. After creating a job, you're ready to upload your data. You upload the record data using the CSV file you created earlier. You can submit up to 100 CSV files as part of a job.

   Request

   curl --location --request PUT 'https://{instance_url}/api/v1/ingest/jobs/913a8b00-7a9e-4331-aaf6-a019ae5021ab/batches
   
   ' \
   --header 'Authorization: Bearer {access_token}' \
   --header 'Content-Type: text/csv' \
   --data-binary '@orders.csv'

   Response

   201 Created

5. Once you're done submitting data, close the job to indicate that the job is ready for processing.

   Request

   curl -v --location --request PATCH 'https://{instance_url}/api/v1/ingest/jobs/913a8b00-7a9e-4331-aaf6-a019ae5021ab' \
   --header 'Authorization: Bearer {access_token}' \
   --header 'Content-Type: application/json' \
   --data-raw '{
   "state" : "UploadComplete"
   }'

   Response

   {
     "object": "Orders",
     "id": "913a8b00-7a9e-4331-aaf6-a019ae5021ab",
     "operation": "upsert",
     "sourceName": "ecomm_api",
     "createdById": "005R0000000zqc8IAA",
     "createdDate": "2021-08-12T17:06:42.921387Z",
     "systemModstamp": "2021-08-12T17:36:42.921387Z",
     "state": "UploadComplete",
     "contentType": "CSV",
     "apiVersion": "v1",
     "contentUrl": "/api/v1/ingest/jobs/913a8b00-7a9e-4331-aaf6-a019ae5021ab/batches"
   }

6. Check the job status and results. To get basic status information on a job, such as the overall job state or the number of records processed, use a Get Job Info request. The job state changes from UploadComplete to either JobComplete or Failed after the files have been loaded into the data lake.

   Request

   curl -v --location --request GET 'https://{instance_url}/api/v1/ingest/jobs/913a8b00-7a9e-4331-aaf6-a019ae5021ab' \
   --header 'Authorization: Bearer {access_token}' \

   Response

   {
     "object": "Orders",
     "id": "913a8b00-7a9e-4331-aaf6-a019ae5021ab",
     "operation": "upsert",
     "sourceName": "GreasyConnector",
     "createdById": "005R0000000zqc8IAA",
     "createdDate": "2021-08-12T17:06:42.921387Z",
     "systemModstamp": "2021-08-12T18:06:42.921387Z",
     "state": "JobComplete",
     "contentType": "CSV",
     "apiVersion": "v1",
     "contentUrl": "/api/v1/ingest/jobs/913a8b00-7a9e-4331-aaf6-a019ae5021ab/batches",
     "retries": 0,
     "totalProcessingTime": 0
   }

See Also

• Data Cloud Reference Guide: Get Started
• Data Cloud Reference Guide: Get Started with Ingestion API

j

No Results

Search Tips:

• Please consider misspellings
• Try different search keywords

Data Cloud Reference Guide

Connect Data

Ingest Data

Ingest Data into Data Cloud

Get Started with Ingestion API

Streaming Ingestion Walkthrough

Bulk Ingestion Walkthrough

Streaming Ingestion

Bulk Ingestion

Real-Time Ingestion

Create an Amazon S3 Data Stream

Rotation of S3 Credentials with Metadata API

Model Data in Data Cloud

Query Data