Get Started with Ingestion API | Data Cloud Reference Guide

Prerequisites

Setup an Ingestion API connector to define the endpoints and payload to ingest data.
Create an Ingestion API data stream to configure ingestion jobs and expose the API for external consumption.
Contact your Data Cloud Admin to get endpoint details configured in Ingestion API Developer Info.

Authentication

Set up a connected app to authenticate and request access to Data Cloud Ingestion API. The connected app enables standard OAuth protocols for authentication and authorization. Follow the instructions in Salesforce Help Create a Connected App, and configure the app as needed. In your connected app, make sure Enable OAuth Settings and necessary OAuth scopes are selected. The possible scopes are:

Access and manage your Data Cloud Ingestion API data (cdp_ingest_api).
Access and manage your data (api).
Perform requests on your behalf at any time (refresh_token, offline_access).

Your orgs must be provisioned with Data Cloud licenses and the users must be assigned to appropriate roles for having full access to objects in the Data Cloud. Refer to User Roles and Permission Sets in Data Cloud before setting up the Connected App.

Important

Acquire a Salesforce Access Token

Send a request for acquiring the Salesforce access token. Here’s how a sample request is going to look like.

1POST https://login.salesforce.com/services/oauth2/token
2Content-Type : x-www-form-urlencoded
3grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer
4&assertion=eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiJ9.eyJpc3MiOiIzTVZHOUFPcDRrYnJpWk9JZmg0eWZPQVlQUjFpYzJqb1JDQkhwZFZtaXBXU05sbXNwT2Vib1VtcWdMdHJIcDlRVFRQMGpWc3g0SHlzUW4xLk1SazFyIiwic3ViIjoic3VkaGVlckBvcmdtYWluMi5jb20iLCJhdWQiOiJodHRwOi8vc3VrdW1hci1sdG0uaW50ZXJuYWwuc2FsZXNmb3JjZS5jb20iLCJleHAiOjE2MDQ2Mzc4MTkyNTd9.WkEcmaGj_jhJpyFqXwEUtw-bkeXYuxkqc2c3lkwbGjg3GDo7sStITht_O_dSNanS0xk2q9ARA6Uom-uaiyNw_GGGK5L3jBQjK-ytrGumuJAsujjfSmLdYStxPVnjU2zyuPSWMQJmeDegxDgmeht3UQyluj0q4lX3rayx0Ol-vZfTuzr4YlxmdOKTNrPJCQ8h26uMSPFrQmmqcssrSg1DRgm3kMh15VKM3yZyNAg43EPw1BfncPWIxOmsGt2JxUUclBER3enTLM2cJKr9fDx16Ssesj2UKqXGbwN68gCF5rZxD6yGN5B0VeZIZVJrY5cTtExqBp3Lj8UOcqUj6esyCQ

Refer to OAuth 2.0 JWT Bearer Flow for Server-to-Server Integration for creating a JWT assertion.

Note

Response Format

This example shows a response from Salesforce.

1{
2"access_token": "00Dxx0000006JFr!AQEAQL7SvQ1t..2yYtyM0jfXtMNriivdMpTYE3mThqATnm5YqPIX.xnzUC45zd0HEIU3tc9GdHLqWOFVprsgszWdYMPRUnOy",
3"scope": "cdp_ingest_api",
4"instance_url": "http://orgmain2.localhost.internal.salesforce.com:6109",
5"id": "http://dcorg-ltm.internal.salesforce.com:6109/id/00Dxx0000006JFrEAM/005xx000001X9hBAAS",
6"token_type": "Bearer"
7}

Exchanging Salesforce Access Token for Data Cloud Access Token

Now that you've acquired the Salesforce access token, use it to get the Data Cloud access token to invoke the Ingestion API.

This example shows a request for Data Cloud access token.

1POST https://a360.cdp.cdp001.dev1-uswest2.aws.sfdc.cl/services/a360/token
2Content-Type : x-www-form-urlencoded
3grant_type=urn:salesforce:grant-type:external:cdp
4&subject_token=00Dxx0000006JFr!AQEAQL7SvQ1t..2yYtyM0jfXtMNriivdMpTYE3mThqATnm5YqPIX.xnzUC45zd0HEIU3tc9GdHLqWOFVprsgszWdYMPRUnOy
5&subject_token_type=urn:ietf:params:oauth:token-type:access_token

This example shows a sample response.

1{
2"access_token": "eyJraWQiOiJDT1JFLjAwRHh4MDAwMDAwNkpGci4xNjAzMjM5NzI2NjI5IiwidHlwIjoiSldUIiwiYWxnIjoiUlMyNTYifQ.eyJzdWIiOiJodHRwOi8vc3VrdW1hci1sdG0uaW50ZXJuYWwuc2FsZXNmb3JjZS5jb206NjEwOS9pZC8wMER4eDAwMDAwMDZKRnJFQU0vMDA1eHgwMDAwMDFYOWhCQUFTIiwiYXVkIjoiYTM2MCIsInNjcCI6ImNkcHByb2ZpbGUgY2RwcXVlcnkiLCJuYmYiOjE2MDQ1NTE3NzYsInNmb2lkIjoiMDBEeHgwMDAwMDA2SkZyIiwiaXNzIjoiaHR0cDovL3N1a3VtYXItbHRtLmludGVybmFsLnNhbGVzZm9yY2UuY29tOjYxMDkvIiwic2Z1aWQiOiIwMDV4eDAwMDAwMVg5aEIiLCJleHAiOjE2MDQ1NTkwMjgsImlhdCI6MTYwNDU1MTgzNiwianRpIjoiYjk1ODA1ZDUtM2MzYS00MDNkLWJiNjgtNjBkNTJlNmM4OTBjIiwic2ZhcHBpZCI6IjNNVkc5QU9wNGticmlaT0lmaDR5Zk9BWVBSMWljMmpvUkNCSHBkVm1pcFdTTmxtc3BPZWJvVW1xZ0x0ckhwOVFUVFAwalZzeDRIeXNRbjEuTVJrMXIifQ.GAWqiPpVJoDig-adGtD4d5cO7L7JaIn2mOYH9GyDD7oO7Ayrsnv_k05QRPp-B1gSd4U8lqE_cuC0MTE9L6lzJA",
3"instance_url": "https://a360.cdp.cdp001.dev1-uswest2.aws.sfdc.cl",
4"token_type": "Bearer",
5"issued_token_type": "urn:ietf:params:oauth:token-type:jwt",
6"expires_in": 7191
7}

Bulk Ingest API Limits

Item	Description
API usage limits	After each request, your app must check the response code. The `HTTP 429 Too Many Requests` status code indicates the app must reduce its request frequency.
Bulk Job Retention Time	Any open bulk jobs with the status of Open or Upload Complete that are older than 7 days, are deleted from the ingestion queue.
Maximum Number of Files per Job	You can upload one file at a time per bulk job. A job can have a maximum of 100 files.
Maximum Payload size	CSV files uploaded via Bulk API have a maximum size of 150 MB.
Number of Requests or Jobs Allowed per Hour	20
Number of Concurrent Requests or Jobs Allowed at One Time	5

Streaming Ingest API Limits

Item	Description
API usage limits	After each request, your app must check the response code. The `HTTP 429 Too Many Requests` status code indicates the app must reduce its request frequency.
Expected Latency	Data is processed asynchronously approximately every 3 minutes.
Maximum Payload Size Per Request	JSON data uploaded via Streaming API have a maximum body size of 200 KB per request
Maximum Number of Records that can be Deleted	You can delete a maximum of 200 records via Streaming API.
Total Number of Requests per Second Across All Ingestion API Object Endpoints	250 requests

Status Codes and Error Responses

HTTP Response Code	Description
200 OK	Request succeeded.
201 Created	Indicates that the resource was successfully created.
202 Accepted	The request was accepted and the data will be processed asynchronously.
204 No Content	Job was successfully deleted.
400 Bad Request	Server can’t process the request due to client error. Possible causes are a malformed request syntax or invalid request body.
401 Unauthorized	Authentication failed because the JWT is invalid or expired. Refresh the token.
404 Not Found	Client error: the requested resource doesn’t exist.
409 Conflict	Client error: unable to update the job state given its status.
429 Conflict	The user has sent too many requests in a given amount of time. Implement a back-off policy to reduce the number of requests.
500 Internal Server Error	Internal server error. Retry the request.

Data Upload Limits

For streaming: Small payloads (up to 200 KB per single request)
For bulk: Large CSVs up to 150 MB

For guidance on overall limitations view Data Cloud Limits and Guidelines.

Note