Data Cloud Reference Guide

Data Cloud Reference Guide
Connect Data
Ingest Data
Ingest Data into Data Cloud
Get Started with Ingestion API
Streaming Ingestion
Bulk Ingestion
Create a Job
Upload Job Data
Close or Abort a Job
Delete a Job
Get Job Info
Get All Jobs
Create an Amazon S3 Data Stream
Rotation of S3 Credentials with Metadata API
Model Data in Data Cloud
Extract Data
PDF

Bulk Ingestion

With the Data Cloud Ingestion API, you can upsert or delete large data sets. Prepare a CSV file for the data you want to upload, create a job, upload job data, and let Salesforce take care of the rest.

Bulk Jobs and Operations

The Ingestion API upserts or deletes data in Data Cloud using jobs. A bulk job typically goes through following stages:
  • Create a job to specify the object type of the data being processed and the operation that’s performed on the data. The operations supported are upserting bulk data or deleting bulk data.
  • After the job is created upload data in CSV format to the job.
  • To signal the data is ready to be processed, close the job. You can choose to abort the job if necessary.
  • Monitor the progress of the job and act on any failed records.
  • Delete a job.

Prepare CSV Files

Lists the field names for the object that you're processing in the first row in the CSV file. Each subsequent row corresponds to a record in your Data Cloud data lake. All the records in a CSV file must be for the same object. You specify this object when you first create the job.
  • Include all required fields when you create a record.
  • Each field-name header in the file must be the same as the Datasource Object's field names. Results only include columns that are a match.
  • Updating records works as a full replace. Patch semantics aren’t supported.
  • Files must be in UTF-8 format. Upload data must not exceed 150 MB.
  • CSV files are expected to be formatted according to RFC 4180, Common Format, and MIME Type for CSV Files.
  • Only supports comma field delimiters.
  • Empty field values are set to null.

Example

Valid Date Format in Records

The Ingestion API supports ISO 8601 UTC with Zulu format.
  • Use the yyyy-MM-dd format to specify date values. For example, in 2021-07-05
    • yyyy is the four-digit year
    • MM is the two-digit month
    • dd is the two-digit day
  • Use the yyyy-MM-ddTHH:mm:ss.SSSZ format to specify dateTime fields. For example, 2021-07-05T09:31:44.457Z
    • yyyy is the four-digit year
    • MM is the two-digit month
    • dd is the two-digit day
    • 'T' is a separator indicating that time-of-day follows
    • HH is the two-digit hour
    • mm is the two-digit minute
    • ss is the two-digit seconds
    • SSS is the optional three-digit milliseconds (000=999)
    • 'Z' is the reference UTC timezone