Bulk Ingestion

With the Data Cloud Ingestion API, you can upsert or delete large data sets. Prepare a CSV file for the data you want to upload, create a job, upload job data, and let Salesforce take care of the rest.

The Ingestion API upserts or deletes data in Data Cloud using jobs. A bulk job typically goes through following stages:

  • Create a job to specify the object type of the data being processed and the operation that’s performed on the data. The operations supported are upserting bulk data or deleting bulk data.
  • After the job is created upload data in CSV format to the job.
  • To signal the data is ready to be processed, close the job. You can choose to abort the job if necessary.
  • Monitor the progress of the job and act on any failed records.
  • Delete a job.

Lists the field names for the object that you're processing in the first row in the CSV file. Each subsequent row corresponds to a record in your Data Cloud data lake. All the records in a CSV file must be for the same object. You specify this object when you first create the job.

  • Include all required fields when you create a record.
  • Each field-name header in the file must be the same as the Datasource Object's field names. Results only include columns that are a match.
  • Updating records works as a full replace. Patch semantics aren’t supported.
  • Files must be in UTF-8 format. Upload data must not exceed 150 MB.
  • CSV files are expected to be formatted according to RFC 4180, Common Format, and MIME Type for CSV Files.
  • Only supports comma field delimiters.
  • Empty field values are set to null.

The Ingestion API supports ISO 8601 UTC with Zulu format.

  • Use the yyyy-MM-dd format to specify date values. For example, in 2021-07-05

    • yyyy is the four-digit year
    • MM is the two-digit month
    • dd is the two-digit day

  • Use the yyyy-MM-ddTHH:mm.SSSZ format to specify dateTime fields. For example, 2021-07-05T09:31:44.457Z

    • yyyy is the four-digit year
    • MM is the two-digit month
    • dd is the two-digit day
    • 'T' is a separator indicating that time-of-day follows
    • HH is the two-digit hour
    • mm is the two-digit minute
    • ss is the two-digit seconds
    • SSS is the optional three-digit milliseconds (000=999)
    • 'Z' is the reference UTC timezone

  • Create a Job

    Creates a job that represents the bulk operation for loading data into the Data Cloud org.

  • Upload Job Data

    Uploads CSV data for the job to process. A job can consist of up to 100 data files. After all of the data has been uploaded, close the job to indicate the data is ready to be processed.

  • Close or Abort a Job

    Close a job by updating the job state to UploadComplete. When you close a job, the uploaded data is enqueued for processing. To abort a job, update the job state to Aborted. If you abort a job, the job doesn’t get queued or processed. Any data uploaded as part of the aborted job is deleted.

  • Delete a Job

    To delete, a job must have a state of UploadComplete, JobComplete, Aborted, or Failed. When you delete a job, the stored job data is also deleted and its metadata is removed.

  • Get Job Info

    Retrieves detailed information about the specified job.

  • Get All Jobs

    Retrieves all jobs in Data Cloud.

See Also