Set Up a Databricks Data Federation Connection

Set up a connection between AWS or Azure hosted Databricks and Data Cloud to access data.

User Permissions Needed 
To create a connection in Data Cloud:System Admin

Before you begin:

  • Review Data Cloud IP Allowlist, and update your Databricks allowlists.
  • Databricks clusters must use Databricks Runtime 13.1 or later and in shared or single-user access mode.
  • Make sure these compute and workspace requirements in Databricks are met: .
    • Workspace requirements - Query data on Databricks from Salesforce: None, Unity Catalog recommended.
    • Compute requirements:
      • Network connectivity from your Databricks Runtime cluster or SQL warehouse to Salesforce. For more information, see Networking recommendations for Lakehouse Federation.
      • Databricks clusters must use Databricks Runtime 13.1 or above and shared or single-user access mode.
      • SQL warehouses must be Pro or Serverless.
    • To set up connectivity for data federation, the account admin must have a Workspace admin role.
    • The Workspace admin must complete these setup tasks.

ClientID/ClientSecret authentication is only supported using Azure Databricks-managed Service Principals. Microsoft Entra ID-managed principals are not supported.

  1. In Data Cloud, go to Data Cloud Setup.

  2. Under Configuration, select Connectors.

  3. Click New.

  4. Under Source, select Databricks and click Next.

  5. Enter a connection name and connection API name.

  6. Select one of the three authentication methods.

    • Select Username & Password. Contact your Databricks administrator to obtain the Databricks username and password.

    • Select Client ID & Client Secret. Contact your Databricks administrator to obtain the client ID and secret.

    • Select Identity Provider Based and complete the setup in the next step.

  7. To use identity provider–based authentication, click Identity Provider Based and copy the generated external ID.

    1. Add a Service Principal identity in the Databricks Admin panel.

      1. In your Databricks workspace, click your user icon on the top right and then click Settings.

      2. In the Settings sidebar, under Workspace admin, click Identity and access.

      3. Against Service principals, click Manage, and then click Add service principal. If it is an Azure managed Databricks instance, then the service principal must be Databricks managed.

      4. In the Add service principal dialog box, click Add new.

      5. Enter a name for the service principal and click Add.

        After the service principal is created, you can see a service principal with the name you gave and an alphanumeric Application ID. We’ll use this Application ID as Client ID in the DataCloud connections window.

    2. Create a Federation Policy and attach it to the service principal that you created.

      1. In the JSON file, for the issuer field, enter your org’s My Domain URL (from Salesforce org) followed by /services/connectors. For example, https://yourcompany.my.salesforce.com/services/connectors.

      2. For the audiences field, enter your org’s My Domain URL. For example, https://yourcompany.my.salesforce.com. See, My Domain.

      3. For the subject field, enter the external ID copied from the Data Cloud new connection dialog box.

      4. To get the Service ID of the service principal, execute this command on the admin console (CLI).

        If you haven't installed the Databricks CLI, see Install or Update the Databricks CLI.

      5. Copy the 15-digit service ID of the service principal and enter it in the service ID field and execute the command shown in the JSON blurb.

    3. Grant permissions for the service principal to access the warehouse.

      1. In the left navigation panel of DBX instance, select SQL Warehouse.

      2. On the SQL warehouses tab, select any of the available warehouses that you can access.

      3. Click Permissions.

      4. In the Type to add multiple users or groups textbox, enter the service principal details.

      5. In the service principal popup, add ‘Can use’ permissions.

      6. Close this window.

    4. Grant permissions for the service principal to access catalog.

      1. In the left navigation panel of the Databricks home, select Catalog.

      2. Select required catalog from the catalog explorer.

      3. Under the Permissions tab, click Grant.

      4. In the Type to add multiple users or groups textbox, enter the service principal details.

      5. Provide appropriate privileges for this service principal. Make sure you have SELECT privileges.

    5. In the Data Cloud connection setup, in the Client Id field, paste the application ID of the service principal that you created.

  8. In the Connection URL field, enter the server hostname of the SQL warehouse along with the port number. For example, adb-8903155206260665.5.azuredatabricks.net:443.

  9. In the Http path field, enter the HTTP path value of the SQL warehouse in Databricks.

  10. To review your configuration, click Test Connection.

  11. Click Save.