Bulk Ingestion

With the Data 360 Ingestion API, you can upsert or delete large data sets. Prepare a CSV file for the data you want to upload, create a job, upload job data, and let Salesforce take care of the rest.

The Ingestion API upserts or deletes data in Data 360 using jobs. A bulk job typically goes through following stages:

  • Create a job to specify the object type of the data being processed and the operation that’s performed on the data. The operations supported are upserting bulk data or deleting bulk data.
  • After the job is created upload data in CSV format to the job.
  • To signal the data is ready to be processed, close the job. You can choose to abort the job if necessary.
  • Monitor the progress of the job and act on any failed records.
  • Delete a job.

Jobs transition through these status values during their lifecycle.

StatusDescription
OpenJob is created and ready to accept data uploads.
UploadCompleteAll data is uploaded, the job is closed and ready for processing.
InProgressThe system is actively processing the uploaded data.
JobCompleteProcessing completed successfully and data is available.
FailedProcessing failed. Check error details.
AbortedJob was manually aborted.

Lists the field names for the object that you’re processing in the first row in the CSV file. Each subsequent row corresponds to a record in your Data 360 data lake. All the records in a CSV file must be for the same object. You specify this object when you first create the job.

  • Include all required fields when you create a record.
  • Each field-name header in the file must be the same as the Datasource Object’s field names. Results only include columns that are a match.
  • Updating records works as a full replace. Patch semantics aren’t supported.
  • Files must be in UTF-8 format. Upload data must not exceed 150 MB.
  • CSV files are expected to be formatted according to RFC 4180, Common Format, and MIME Type for CSV Files.
  • Only supports comma field delimiters.
  • Empty field values are set to null.

The Ingestion API supports ISO 8601 UTC with Zulu format.

  • Use the yyyy-MM-dd format to specify date values. For example, in 2021-07-05:

    • yyyy is the four-digit year
    • MM is the two-digit month
    • dd is the two-digit day

  • Use the yyyy-MM-dd'T'HH:mm:ss.SSS'Z' format to specify dateTime fields. For example, in 2021-07-05'T'09:31:44.457'Z':

    • yyyy is the four-digit year
    • MM is the two-digit month
    • dd is the two-digit day
    • 'T' is a separator indicating that time-of-day follows
    • HH is the two-digit hour
    • mm is the two-digit minute
    • ss is the two-digit seconds
    • SSS is the optional three-digit milliseconds (000-999)
    • 'Z' is the reference UTC timezone

  • Create a Job

    Creates a job that represents the bulk operation for loading data into the Data 360 org.

  • Upload Job Data

    Uploads CSV data for the job to process. A job can consist of up to 100 data files. After all of the data has been uploaded, close the job to indicate the data is ready to be processed.

  • Close or Abort a Job

    Close a job by updating the job state to UploadComplete. When you close a job, the uploaded data is enqueued for processing. To abort a job, update the job state to Aborted. If you abort a job, the job doesn’t get queued or processed. Any data uploaded as part of the aborted job is deleted.

  • Delete a Job

    To delete, a job must have a state of UploadComplete, JobComplete, Aborted, or Failed. When you delete a job, the stored job data is also deleted and its metadata is removed.

  • Get Job Info

    Retrieves detailed information about the specified job.

  • Get All Jobs

    Retrieves all jobs in Data 360.

See Also