Set Up a Google Cloud Storage Connection
After you create your Data Cloud instance, you can set up a Google Cloud Storage (GCS) connection. You can create up to five GCS connections. A connection is defined as the unique combination of the bucket name and parent directory.
User Permissions Needed | |
---|---|
To create a connection: | System Administrator |
Before you begin:
- Set Up Permissions for the Google Cloud Storage Connector
- Review the IP addresses to make sure that the GCS connection has the necessary access
-
In Data Cloud, go to Data Cloud Setup.
-
Under Configuration, select More Connectors.
-
Click New.
-
Under Source, select Google Cloud Storage and click Next.
-
Enter a connection name and connection API name.
-
Enter the connection details.
Field Label Description Access Key Programmatic username for API access to Google Cloud Storage bucket. Secret Key Programmatic password for API access to Google Cloud Storage bucket. -
Enter the authentication details.
All folders under the parent directory are migrated to a staging environment where you can select the objects to import into Data Cloud. Because data access charges can apply, it’s recommended that the parent directory and its subfolders contain only essential files.
Field Label Description Bucket Name Specific public cloud storage resource in Google Cloud Service. Parent Directory Parent folder that corresponds to the GCS bucket. The root directory can’t serve as the parent directory. -
To review your configuration, click Test Connection.
-
Click Save.
After the connection details are accepted, Data Cloud begins syncing data between your GCS instance and the Data Cloud staging environment. The sync is designed to mirror the structure of your GCS bucket, so if objects are deleted, they’re also deleted from the staging environment. Your Data Aware Specialist can then create data streams for any objects under the parent directory.
Expect some latency for the initial sync. After the Day 0 load, incremental sync tasks run hourly.