Create an Azure Storage Data Stream
Create a data stream and import objects from Azure Storage. There can be some latency between creating your data stream and object data availability in Data Cloud while data is transferred to the staging environment. The data stream detects the schema from your source file and displays the appropriate Data Cloud data type.
Complete these prerequisites:
- Make sure the Azure Blob Storage Connection is set up. For more information, see See Set Up an Azure Connection.
-
In Data Cloud, go to the Data Streams tab and click New.
You can also use App Launcher to find and select Data Streams.
-
Select Microsoft Azure Blob Storage.
-
Select the applicable connector from the Connection dropdown.
-
Complete file and source details.
Field Label Description File Type Select CSV or Parquet. For more information, see Supported File Formats and Delimiters. Import From Directory The remaining folder path under the container that points to a file’s specific location. Place your source files in the directory because the data stream can’t recognize files stored in nested subdirectories. File Name Name of the file that must be retrieved from the specified folder. The field is pre-populated with '*'. If no file is specified, the system chooses the first file found. After you create a data stream, it retrieves all files found in the directory. Wildcards are also supported, for example you can use *abc*.csv to indicate retrieval of all files containing “abc” in their name. Each time the stream runs, all files satisfying the wildcard are imported. Source A label designating the external system from where data is sourced. Multiple data streams can use the same label for the Source. -
Complete Object Details. You can create a data lake object (DLO) or choose to use an existing DLO.
If you choose to create a DLO then refer to Naming standards for data lake objects. If you choose to use an existing DLO then refer to Using existing data lake object to create a data stream and familiarize yourself with the guardrails to be considered when using an existing DLO.
-
Select a data stream category and primary key. Add new formula fields if needed.
-
From the Data Space dropdown, select the applicable data space or the default data space.
-
Fill in deployment details and click Deploy.
Your data stream is created. Map your data stream to data model objects and start using your data for your Azure Storage use cases.
Any privacy or compliance requests processed in Data Cloud only operate on the data within your Data Cloud org. Records containing personal data in external products like Azure Storage aren’t affected. For example, a consumer asserts their right to be forgotten and requests that you delete any copies of personal data. This consumer has records in Azure Storage and a Data Cloud profile. If you’ve enabled the Azure Storage Service Connector, you must submit deletion jobs to both Azure Storage and Data Cloud.