Data Cloud Developer Guide
jGet Started with Ingestion API
Before using APIs in Data Cloud, complete the prerequisites, set up
authentication, and know the limits that apply to Bulk Ingest API and Streaming Ingest
API.
Prerequisites
- Setup Ingestion API connector: defines the endpoint and payload to consume the data.
- Create and Deploy Data Stream: configures ingestion jobs and exposes the API for external consumption.
- Configure Connected App: enables external applications to integrate with Salesforce using OAuth.
- Request Data Cloud access token from Salesforce OAuth Authorization Flows: implements OAuth dance to obtain and refresh access token required for making API requests.
Authentication
The Data Cloud Ingestion APIs are a set of RESTful APIs that support all OAuth 2.0 flows supported by other Salesforce REST APIs. In order to load data via the APIs, your connected app requires these scopes:
- Access and manage your Data Cloud Ingestion API data (cdp_ingest_api).
- Access and manage your data (api).
- Perform requests on your behalf at any time (refresh_token, offline_access).
After your connected app has the proper scopes, request an access token from Salesforce and exchange it for a Data Cloud access token. The access token is sent in an Authorization header using the Bearer schema.
Valid Date Format in Records
The Ingestion API supports ISO
8601 UTC with Zulu format.
- Use the yyyy-MM-dd format to specify date values. For example, in 2021-07-05
- yyyy is the four-digit year
- MM is the two-digit month (01-12)
- dd is the two-digit day (01-31)
- Use the yyyy-MM-ddTHH:mm:ss.SSSZ format to specify dateTime fields. For example,
2021-07-05T09:31:44.457Z
- yyyy is the four-digit year
- MM is the two-digit month (01-12)
- dd is the two-digit day (01-31)
- 'T' is a separator indicating that time-of-day follows
- HH is the two-digit hour (00-23)
- mm is the two-digit minute (00-59)
- ss is the two-digit seconds (00-59)
- SSS is the optional three-digit milliseconds (000-999)
- 'Z' is the reference UTC timezone
Ingest Bulk API Limits
| Item | Description |
|---|---|
| API usage limits | After each request, your app must check the response code. The HTTP 429 Too Many Requests status code indicates the app must reduce its request frequency. |
| Bulk Job Retention Time | Any open bulk jobs with the status of Open or Upload Complete that are older than 7 days, are deleted from the ingestion queue. |
| Maximum Number of Files per Job | You can upload one file at a time per bulk job. A job can have a maximum of 100 files. |
| Maximum Payload size | CSV files uploaded via Bulk API have a maximum size of 150 MB. |
| Number of Requests or Jobs Allowed per Hour | 20 |
| Number of Concurrent Requests or Jobs Allowed at One Time | 5 |
Ingest Streaming API Limits
| Item | Description |
|---|---|
| API usage limits | After each request, your app must check the response code. The HTTP 429 Too Many Requests status code indicates the app must reduce its request frequency. |
| Expected Latency | Data is processed asynchronously approximately every 3 minutes. |
| Maximum Payload Size Per Request | JSON data uploaded via Streaming API have a maximum body size of 200 KB per request |
| Maximum Number of Records that can be Deleted | You can delete a maximum of 200 records via Streaming API. |
| Total Number of Requests per Second Across All Ingestion API Object Endpoints | 250 requests |
Status Codes and Error Responses
| HTTP Response Code | Description |
|---|---|
| 200 OK | Request succeeded. |
| 201 Created | Indicates that the resource was successfully created. |
| 202 Accepted | The request was accepted and the data will be processed asynchronously. |
| 204 No Content | Job was successfully deleted. |
| 400 Bad Request | Server can’t process the request due to client error. Possible causes are a malformed request syntax or invalid request body. |
| 401 Unauthorized | Authentication failed because the JWT is invalid or expired. Refresh the token. |
| 404 Not Found | Client error: the requested resource doesn’t exist. |
| 409 Conflict | Client error: unable to update the job state given its status. |
| 429 Conflict | The user has sent too many requests in a given amount of time. Implement a back-off policy to reduce the number of requests. |
| 500 Internal Server Error | Internal server error. Retry the request. |
Data Upload Limits
- For streaming: Small payloads (up to 200 KB per single request)
- For bulk: Large CSVs up to 150 MB