Data Cloud Reference Guide

Data Cloud Reference Guide
Connect Data
Ingest Data
Ingest Data into Data Cloud
Get Started with Ingestion API
Streaming Ingestion Walkthrough
Bulk Ingestion Walkthrough
Streaming Ingestion
Bulk Ingestion
Create an Amazon S3 Data Stream
Rotation of S3 Credentials with Metadata API
Model Data in Data Cloud
Extract Data
PDF

Bulk Ingestion Walkthrough

This walkthrough guides you through the steps for loading records using bulk ingestion.
Before you can start, make sure you’ve completed the prerequisites required to set up your Ingestion API:
  • Set up an Ingestion API connector to define the endpoints and payload to ingest data.
  • Download the object endpoints from the configured Ingestion API connector. The object endpoints and the Ingestion API connector name are used as parameters in the API calls.
  • Create an Ingestion API Data Stream to configure ingestion jobs and expose the API for external consumption.
  1. Create a CSV file that has a header row matching the fields in the data stream you created. Each file must not exceed 150 MB. Here's some sample data of an orders.csv file used in this walkthrough.
  2. Request a Data Cloud access token. For more information, refer to Authentication. The access_token property in the token exchange response contains the bearer token to use for the authorization header. The instance_url is the Data Cloud instance where the Ingestion API is hosted.
  3. To upload the data, first create a job. In this example, we’re upserting orders data from an ecomm Ingestion API connector. If you’re deleting records, use the delete operation instead. You must get a JSON response that includes the job id, with an Open job state. You’ll need the job id in the next steps.
    Request
    Response
  4. After creating a job, you're ready to upload your data. You upload the record data using the CSV file you created earlier. You can submit up to 100 CSV files as part of a job.
    Request
    Response
  5. Once you're done submitting data, close the job to indicate that the job is ready for processing.
    Request
    Response
  6. Check the job status and results. To get basic status information on a job, such as the overall job state or the number of records processed, use a Get Job Info request. The job state changes from UploadComplete to either JobComplete or Failed after the files have been loaded into the data lake.
    Request
    Response