Customer Data Platform Developer Guide

Customer Data Platform Developer Guide
Connect Data
Ingest Data
Load Data into Customer Data Platform
Get Started with Ingestion API
Streaming Ingestion
Bulk Ingestion
Create an Amazon S3 Data Stream
Model Data
Extract Data
PDF

Load Data into Customer Data Platform

The Ingestion API provides a RESTful interface that supports both streaming and bulk interaction patterns for programmatically loading data into Customer Data Platform. A single API connector can be used for performing both bulk and streaming uploads.

What’s the Difference between Bulk and Streaming Ingest APIs?

If you're wondering about which API to use, it helps if you consider where the data is coming from and the problem you’re trying to solve.

Bulk API
  • Moving large amounts of data on a daily, weekly, or monthly schedule.
    • Legacy systems where you can only export data in off-peak hours.
    • A new Customer Data Platform org that you want to backfill with 30/60/90/X days of data.
Streaming API
  • Small micro-batches of records being updated in near-real time.
    • Source system built on modern streaming architectures.
    • Change data capture events.
    • Consuming data from webhooks.

We recommend you familiarize yourself with Ingestion API Data Flow to get a better understanding on setting up and configuring Ingestion API. Contact your CDP Admin to get details specified in Ingestion API Developer Info.

Note

Example

Let’s imagine you’re an integrator working for Northern Trail Outfitters (NTO). You’ve been tasked with extracting the runner profiles and activity logs from NTO’s Track My Run mobile app and loading them into Customer Data Platform. The marketer you’re working with has indicated that he needs the last 90 days of historical data and wants all new and updated data on a go-forward basis.

You query the last 90 days from your data warehouse as requested and you chunk up the data into 150 MB CSV files. You then load the data into Customer Data Platform via the Bulk API, understanding that the Bulk API is an appropriate solution for one-time operations. Now that the initial data set is loaded, you decide it’s best to synchronize future changes with an upsert operation in the streaming API. These updates can be forwarded to Customer Data Platform in micro-batches as soon as the data becomes available.