Set Up an Amazon Kinesis Connection

Create an Amazon Kinesis connection to pull data from Kinesis data streams into Data Cloud. To connect to Data Cloud, you can use either key-based authentication or identity-provider (IdP) authentication by configuring Salesforce as an OIDC-based IdP in AWS. Using Salesforce as an IdP provides a more secure and robust role-based authentication.

User Permissions Needed 
To create an Amazon Kinesis connection:System Admin profile or Data Cloud Architect permission set

Before you begin:

  1. In Data Cloud, click Setup, and select Data Cloud Setup.

  2. Under External Integrations, select Other Connectors and click New.

  3. On the Source tab, select Amazon Kinesis and click Next.

  4. Enter a connection name and connection API name.

  5. If you want to use key-based authentication, click Access Key/Secret Based and enter the information for the IAM user connecting to the Kinesis data stream.

    If you are using key-based authentication, skip the next step and continue configuring your Kinesis data stream.

  6. If you want to use identity provider-based authentication, click Identity Provider Based.

    1. In the AWS IAM dashboard, select Identity providers, and click Add provider.

    2. Enter the required information and then click Add Provider.

      ItemWhat to EnterExample
      Provider typeOpenID Connect
      Provider URLMy Domain URL/services/connectors>https://yourcompany.my.salesforce.com/services/connectors
      Audience<My Domain URLhttps://yourcompany.my.salesforce.com

    3. In AWS IAM create a role to provide access to IAM users.

      1. For Trusted Entity Type, select Web Identity.

      2. Select the provider and audience that you created.

      3. Add a JSON permission policy document for the role that you are creating. Use this JSON blurb to define the required permissions.

      4. Enter a name for the role, and click Create role.

    4. From Data Cloud, copy the External ID.

    5. On the IAM role page’s Trust relationship tab, click Edit trust policy.

    6. In the JSON blurb, replace {external_app_id} with the Data Cloud External ID.

    7. From AWS IAM, copy the role’s Amazon Resource Name.

    8. In Data Cloud, enter the Amazon Resource Name in the IAM Role Name field.

  7. Enter your Kinesis data stream's name.

  8. Enter the name of the AWS region where your Kinesis data stream is hosted.

  9. Enter the Kinesis data stream’s endpoint. See Amazon Kinesis data streams endpoints.

  10. To review your configuration, click Test Connection.

  11. When the connection succeeds, click Save.

  12. Upload a YAML file that describes the schema of your Kinesis data stream object and complies with the OpenAPI specification.

The file must define only one object, and that sole object’s schema must be flat.

After the connector details are accepted, the connection is created and ready to use. It’s listed under More Connectors. A Data Aware Specialist can now create data streams.

Considerations

  • YAML vs. JSON: While the schema of the object is defined in the YAML mentioned above, each record (specifically, the data field in each record) in your Kinesis data stream must be a flat JSON object.
  • Connection Scope: Only one connection can be created per Kinesis data stream which is uniquely identified by its name and region.
  • Schema Registry: The Kinesis connector doesn't support connecting to a schema registry. If you want to evolve the schema of your Kinesis data stream's object, you need to upload a new YAML file with additive changes (existing fields can't be removed). Any record whose schema varies from the one registered with Data Cloud will fail to be ingested.