Walkthrough for Streaming Insert

This walkthrough guides you through the steps for loading records using the Streaming API.

Before you can start, make sure you’ve completed the prerequisites required to set up your Ingestion API:

Setup Ingestion API connector
Create and Deploy Data Stream
Configure connected app

Create a collection of JSON objects that matches the schema of the source objects you defined as part of your data stream. Wrap the collection in a data envelope. The payload must be less than 200 KB. For example, orders.json:

1{
2       "data": [{
3       "id": 1,
4       "contact_name": "Joe Smith",
5       "created_date": "2021-07-22T08:57:08.118Z",
6       "tax_exempt": false,
7       "ship_address": "9818 Main",
8       "total": 77506,
9       "tax_rate": 84,
10       "modified_date": "2021-07-05T09:31:44.457Z"
11       },
12       {
13       "id": 2,
14       "contact_name": "Tina Smith",
15       "created_date": "2021-07-08T22:57:02.008Z",
16       "tax_exempt": false,
17       "ship_address": "245 Main",
18       "total": 16986,
19       "tax_rate": 61,
20       "modified_date": "2021-07-03T06:20:52.886Z"
21       }
22]
23}

Request a Customer Data Platform access token. The access_token property in the token exchange response contains the bearer token to use for the authorization header. The instance_url is the Customer Data Platform instance where the API is hosted.
Use the Synchronous Record Validation method and validate that your ingestion request is configured correctly. Requests that fail validation aren’t processed into the data lake. If the request fails validation, the API returns a report indicating which records are problematic and why. Fix the request based on the validation feedback and resubmit until you received a status code in the 200 s from the API.
1curl --location --request POST 'https://{instance_url}/api/v1/ingest/sources/ecomm/Order/actions/test' \ 2--header 'Authorization: Bearer {access_token}' \ 3--header 'Content-Type: application/json' \ 4--data @orders.json
After you’re confident the integration is properly configured, upload your JSON data via the Streaming API. You should receive a 202 Accepted response indicating your data has been accepted for processing.
1curl --location --request POST 'https://{instance_url}/api/v1/ingest/sources/ecomm/Order/' \ --header 'Authorization: Bearer {access_token}' \ 2--header 'Content-Type: application/json' \ 3--data @orders.json

Customer Data Platform Developer Guide

Walkthrough for Streaming Insert