Set Up an Amazon Kinesis Connection

Create an Amazon Kinesis connection to pull data from Kinesis data streams into Data 360. To connect to Data 360, you can use either key-based authentication or identity-provider (IdP) authentication by configuring Salesforce as an OIDC-based IdP in AWS. Using Salesforce as an IdP provides a more secure and robust role-based authentication.

User Permissions Needed
To create an Amazon Kinesis connection:	System Admin profile or Data Cloud Architect permission set

Before you begin:

Review the VPC interface endpoints to make sure the Amazon Kinesis connection has the necessary access.

In Data Cloud, click Setup, and select Data Cloud Setup.
Under External Integrations, select Other Connectors and click New.
On the Source tab, select Amazon Kinesis and click Next.
Enter a connection name and connection API name.
If you want to use key-based authentication, click Access Key/Secret Based and enter the information for the IAM user connecting to the Kinesis data stream.

If you are using key-based authentication, skip the next step and continue configuring your Kinesis data stream.

If you want to use identity provider-based authentication, click Identity Provider Based.

In the AWS IAM dashboard, select Identity providers, and click Add provider.

Enter the required information and then click Add Provider.

Item	What to Enter	Example
Provider type	OpenID Connect
Provider URL	My Domain URL/services/connectors>	https://yourcompany.my.salesforce.com/services/connectors
Audience	<My Domain URL	https://yourcompany.my.salesforce.com

In AWS IAM create a role to provide access to IAM users.

For Trusted Entity Type, select Web Identity.
Select the provider and audience that you created.

Add a JSON permission policy document for the role that you are creating. Use this JSON blurb to define the required permissions.

1{"Version": "2012-10-17"
2
3    "Statement": [
4    {
5    "Effect": "Allow",
6    "Action": [
7    "kinesis:SubscribeToShard",
8    "kinesis:DescribeStreamSummary",
9    "kinesis:ListShards",
10    "kinesis:DescribeStreamConsumer",
11    "kinesis:GetShardIterator",
12    "kinesis:GetRecords",
13    "kinesis:ListStreamConsumers",
14    "kinesis:RegisterStreamConsumer",
15    "kinesis:DeregisterStreamConsumer"
16    ],
17    "Resource": [
18    "arn:aws:kinesis:*:<account-id>:stream/<kinesis-stream-name>",
19    "arn:aws:kinesis:*:<account-id>:stream/<Kinesis-stream-name>/*"
20    ]
21    }
22    ]
23    }

Enter a name for the role, and click Create role.

From Data Cloud, copy the External ID.
On the IAM role page’s Trust relationship tab, click Edit trust policy.

In the JSON blurb, replace {external_app_id} with the Data Cloud External ID.

1"Version": "2012-10-17",
2
3"Statement": [
4{
5"Effect": "Allow",
6"Principal": {
7"Federated": "arn:aws:iam::240852588578:oidc-provider/yourcompany.my.salesforce.com/services/connectors/services/connectors"
8},
9"Action": "sts:AssumeRoleWithWebIdentity",
10"Condition": {
11"StringEquals": {
12"yourcompany.my.salesforce.com/services/connectors:sub": "{external_app_id}"
13}
14}
15}
16]

From AWS IAM, copy the role’s Amazon Resource Name.
In Data Cloud, enter the Amazon Resource Name in the IAM Role Name field.

Enter your Kinesis data stream’s name.
Enter the name of the AWS region where your Kinesis data stream is hosted.
Enter the Kinesis data stream’s endpoint. See Amazon Kinesis data streams endpoints.
To review your configuration, click Test Connection.
When the connection succeeds, click Save.
Upload a YAML file that describes the schema of your Kinesis data stream object and complies with the OpenAPI specification.

The file must define only one object, and that sole object’s schema must be flat.

Note

After the connector details are accepted, the connection is created and ready to use. It’s listed under More Connectors. A Data Aware Specialist can now create data streams.

Considerations

YAML vs. JSON: While the schema of the object is defined in the YAML mentioned above, each record (specifically, the data field in each record) in your Kinesis data stream must be a flat JSON object.
Connection Scope: Only one connection can be created per Kinesis data stream which is uniquely identified by its name and region.
Schema Registry: The Kinesis connector doesn’t support connecting to a schema registry. If you want to evolve the schema of your Kinesis data stream’s object, you need to upload a new YAML file with additive changes (existing fields can’t be removed). Any record whose schema varies from the one registered with Data 360 will fail to be ingested.

Next Step

Create an Amazon Kinesis Data Stream