Set Up a Google Cloud Storage Connection

After you create your Data 360 instance, you can set up a Google Cloud Storage (GCS) connection. The GCS connector is bidirectional: set up a source connection to ingest data into Data 360, or set up a target connection to activate data from Data 360 to your GCS bucket. You can create up to five GCS connections. A connection is defined as the unique combination of the bucket name and parent directory.

User Permissions Needed
To create a connection:System Admin profile or Data Cloud Architect permission set

Before you begin:

  • Prepare Your Google Cloud Storage Connection, which includes creating a bucket and folder, generating interoperability (HMAC) keys, and assigning the service account roles required for your connection type
  • Review the IP addresses to make sure that the GCS connection has the necessary access

Set up a source connection to ingest data from your GCS bucket into Data 360.

  1. In Data Cloud, go to Data Cloud Setup.

  2. Under Configuration, select Other Connectors.

  3. Click New.

  4. Under Source, select Google Cloud Storage and click Next.

  5. Enter a connection name and connection API name.

  6. Enter the connection details.

    Field LabelDescription
    Access KeyProgrammatic username for API access to Google Cloud Storage bucket.
    Secret KeyProgrammatic password for API access to Google Cloud Storage bucket.

  7. Enter the authentication details.

    All folders under the parent directory are migrated to a staging environment where you can select the objects to import into Data 360. Because data access charges can apply, it’s recommended that the parent directory and its subfolders contain only essential files.

    Field LabelDescription
    Bucket NameSpecific public cloud storage resource in Google Cloud Service.
    Parent DirectoryParent folder that corresponds to the GCS bucket. The root directory can’t serve as the parent directory.

  8. To review your configuration, click Test Connection.

  9. Click Save.

After the connection details are accepted, Data 360 begins syncing data between your GCS instance and the Data 360 staging environment. The sync is designed to mirror the structure of your GCS bucket, so if objects are deleted, they’re also deleted from the staging environment. Your Data Aware Specialist can then create data streams for any objects under the parent directory.

Expect some latency for the initial sync. After the Day 0 load, incremental sync tasks run hourly.

Set up a target connection to activate data from Data 360 to your GCS bucket.

  1. In Data Cloud, go to Data Cloud Setup.

  2. Under Configuration, select Other Connectors.

  3. Click New.

  4. Select Google Cloud Storage as the data source.

  5. Select the Target tab.

  6. Enter the connection details.

    Field LabelDescription
    Connection NameA unique name for the connection.
    Connection API NameAPI name for the connection.
    Bucket NameName of your Google Cloud Storage bucket.
    Secret KeyHMAC secret key of the service account.
    Access KeyHMAC access key of the service account.
    Parent DirectoryFolder name, including a trailing slash. For example, data cloud/.

  7. To verify connectivity, click Test Connection.

    This test verifies the credentials and bucket access, but it doesn’t verify the folder path.

  8. If the test succeeds, click Save.

After you set up a target connection, create an activation target so that you can send data from Data 360 to GCS.

  1. In Data 360, go to the Activation Target tab and click New.

  2. Select the Google Cloud Storage connection that you configured.

  3. Enter a name for the activation target.

    The Bucket Name and Parent Directory fields auto-populate from the connection.

  4. Click Next, configure the settings based on your use case, and click Save.

You can now attach this activation target to an activation to begin sending data to GCS.

Create a Google Cloud Storage Data Stream