Skip Navigation
Add the #DF24 Developer Keynote to your agenda. Join us in-person on 9/18 at 2:30 p.m. PT or on Salesforce+ at 5 p.m. PT for the must-see session built just for developers.

Data Cloud Reference Guide

Clouds with binary code floating aboveCloud with binary code floating above

No Results

Search Tips:

  • Please consider misspellings
  • Try different search keywords
Data Cloud Reference Guide
Connect Data
Ingest Data
Ingest Data into Data Cloud
Get Started with Ingestion API
Streaming Ingestion
Bulk Ingestion
Create an Amazon S3 Data Stream
Rotation of S3 Credentials with Metadata API
Model Data in Data Cloud
Extract Data
PDF

Ingest Data into Data Cloud

The Ingestion API provides a RESTful interface that supports both streaming and bulk interaction patterns for loading data into Data Cloud. You can use a single Ingestion API connector for performing both bulk and streaming uploads.

When do you use Bulk and Streaming Ingestion?

Consider the data source and the problem you’re trying to solve when choosing the ingestion mode.

Use bulk ingestion when moving large amounts of data on a daily, weekly, or monthly schedule. A few example scenarios are:
  • Moving large amounts of data on a daily, weekly, or monthly schedule.
  • Legacy systems where you can only export data in off-peak hours.
  • A new Data Cloud org that you want to backfill with 30, 60, 90, or X days of data.
Use streaming ingestion for small micro-batches of records being updated in near-real time, such as:
  • Source system built on modern streaming architectures.
  • Change data capture events.
  • Consuming data from webhooks.
Here's an example use case to understand it better.

Example

Imagine you’re an integrator working for Northern Trail Outfitters (NTO). You need to extract the runner profiles and activity logs from NTO’s Track My Run mobile app and load them into Data Cloud. The marketer you’re working with has indicated that he needs the last 90 days of historical data and wants all new and updated data on a go-forward basis.

You query the last 90 days from your data warehouse as requested and you chunk up the data into 150 MB CSV files. You then load the data into Data Cloud via the bulk ingestion, understanding that the bulk ingestion is an appropriate solution for one-time operations. Now that the initial dataset is loaded, you decide it’s best to synchronize future changes with an upsert operation via streaming ingestion. These updates can be forwarded to Data Cloud in micro-batches as soon as the data becomes available.