Set Up an Apache Iceberg File Federation Connection (Beta)

Set up the Apache Iceberg File Federation connection to start the flow of data into Data Cloud.

User Permissions Needed 
To create a connection:System Administrator

Before you begin:

  • Verify your admin has enabled firewalls on the system you want Data Cloud to connect to by including these IP addresses to your allowlists.
  • Check if the Iceberg Catalog that implements the Iceberg REST Open API specification is accessible to Data Cloud.
  • Make sure that the data files in storage must be formatted according to Apache Parquet.
  • Verify that either AWS S3 or ADLS Gen 2 is used for storage.
  • Understand the following compatibilities and support:
    • Cross-region calls in AWS are supported if the client.region property is included in the REST response generated by Iceberg.
    • Iceberg tablets configured with Iceberg V2 MOR Positional Deletes or Equality Deletes are not supported.
    • Iceberg views cannot be read.

  1. In Data Cloud, click Setup, and select Data Cloud Setup.

  2. Under External Integrations, select Other Connectors.

  3. Click New.

  4. On the Source tab, select Apache Iceberg File Federation and click Next.

  5. Enter a connection name, connection API name, and provide the authentication and connection details.

  6. Enter the authentication details.

    1. Choose the storage type. Supported options are S3, Catalog Provided, and Azure.
    2. Enter the Personal Access Token. It's used to authenticate access to resources and APIs.
    3. Enter other relevant authentication details based on the storage type.
Storage Type Authentication Details
S3 Bucket Name -​​ Specific public cloud storage resource in AWS
Access Key - Programmatic username for API access to AWS
Secret Key - Programmatic password for API access to AWS
Note: Provided keys must have the necessary read permissions for Tables metadata and data.
Azure Personal Access Token - used to authenticate access to resources and APIs
Storage Account Name - Use the name of the Azure blob storage account
SAS Token - A Shared Access Signature (SAS) token to grant access to Azure Storage services
  1. Enter the catalog endpoint under Connection Details. Catalog Endpoint is the REST API endpoint that exposes the Iceberg catalog.

Make sure the catalog endpoint is an HTTPS endpoint. Only RESTCatalog is supported and must be compliant with Iceberg REST specification.

  1. To review your configuration, click Test Connection.

  2. Click Save.

After the connector details are accepted, the connection is created and listed under Connectors. You can now create data streams.