Set Up an Amazon MSK Connection

Create an Amazon Managed Streaming Kafka connection to pull data from Amazon MSK data streams into Data Cloud.

User Permissions Needed 
To create an Amazon Managed Streaming Kafka connection:System Admin profile or Data Cloud Architect permission set

Requirements:

  • Firewall: If the MSK instance you want to connect to is behind a network firewall, add these Data Cloud IP addresses to your allowlist before creating a connection.

  1. In Data Cloud, click Setup, and select Data Cloud Setup.

  2. Under External Integrations, select Other Connectors.

  3. Click New.

  4. On the Source tab, select Amazon MSK and click Next.

  5. Enter a connection name and a connection API name.

  6. If you want to privately connect to the Amazon MSK instance and a Private Network Route (PNR) for the instance has not been configured, see Add a Private Network Route for Amazon MSK. If a PNR has already been created, toggle Use Private Network Route and select the appropriate PNR.

  7. If you want to use key-based authentication, click Access Key/Secret based and enter the necessary information.

    If you’re using key-based authentication, skip the next step and continue configuring your Kafka data stream.

  8. If you want to use identity provider-based authentication, click Identity Provider Based.

    1. In the AWS IAM dashboard, select Identity providers, and click Add provider.

    2. Enter the required information and then click Add Provider.

      ItemWhat to EnterExample
      Provider typeOpenID Connect
      Provider URLMy Domain URL/services/connectorshttps://yourcompany.my.salesforce.com/services/connectors
      AudienceMy Domain URLhttps://yourcompany.my.salesforce.com

    3. In AWS IAM create a role to provide access to IAM users.

      1. For Trusted Entity Type, select Web Identity.

      2. Select the provider and audience that you created.

      3. Create a JSON permission policy document for the role that you are creating. Use this JSON blurb to define the required permissions.

      4. Enter a name for the role, and click Create role.

    4. From Data Cloud, copy the External ID.

    5. On the IAM role page’s Trust relationship tab, click Edit trust policy.

    6. In the JSON blurb, replace {external_app_id} with the Data Cloud External ID.

    7. From AWS IAM, copy the role’s Amazon Resource Name.

    8. In Data Cloud, enter the Amazon Resource Name in the IAM Role Name field.

  9. If the Use Private Network Route option is not toggled, specify a comma-delimited list of MSK bootstrap servers. An MSK bootstrap server serves as an entry point for a client to connect to an MSK cluster. See Get the bootstrap brokers using the AWS Management Console

  10. To review your configuration, click Test Connection.

  11. Click Save.

  12. Upload a schema using the Upload Files button in the Schema section of the connector.

  13. Verify the schema in the Preview Schema window and click Save.

After the connector details are accepted, the connection is created and ready to use. It’s listed under Other Connectors.

A Data Aware Specialist can now create data streams.

Considerations

  • YAML vs. JSON: While the schema of the objects are defined in the YAML mentioned above, each record in your MSK data stream must be a flat JSON object.

  • Schema Registry: The MSK connector doesn't support connecting to a schema registry, for example AWS Glue. To evolve the schema of your MSK data streams, upload a new YAML file with only additive changes (no field removals). Records with schemas differing from the registered one in Data Cloud won't be ingested.