Create an Unstructured Data Connection from Azure Storage

Connect unstructured data from Azure Storage for use in your Agentforce, RAG, automation, and analytics workflows. First create an unstructured data lake object (UDLO) in Data 360 to reference the unstructured data. Then create a file notification pipeline to keep your UDLO up to date.

See the Unstructured Data File Formats and Connectors for a list of supported file formats for unstructured data.

To ensure your unstructured data is properly connected to Data 360, first perform the steps to connect unstructured data from your external blob store, then set up file notifications, and finally put data in your external blob store.

Important

Create a UDLO for Unstructured Data from Azure Storage

Create a UDLO in Data 360 to reference unstructured data from Azure Storage.

User Permissions Needed
To connect unstructured data from an external blob store:	Data Cloud Architect

Assign the Data Cloud Architect permission set to the user creating the UDLO. Don’t also assign Data Cloud User to the same user. Assigning both removes Data Cloud from the App Launcher, which prevents you from opening the Data Cloud app to complete and verify these steps.

Note

Before you begin:

Make sure you’ve set up a connection to Azure Storage.

From App Launcher, select Data Cloud.
Click Data Lake Objects and then click New.
From the New Data Lake Object menu, select From External Files, and click Next.
Choose the Azure Storage connector, and click Next.
From the Select Connection dropdown list, select a connection. Data Cloud auto-populates the source based on the connection that you select.
In the Directory field, point to a specific folder or an entire directory in your blob store. All folders and subfolders in a directory are included. Optionally, use wildcard characters to specify a file name pattern for multiple files.
- A directory value is required in two places: the connection’s Container Path and this UDLO Directory field. Neither can be saved empty. Data Cloud resolves the lookup path by concatenating the two: <Container Path>/<Directory>. Split your blob prefix across the two fields so that the concatenation reconstructs the real path exactly once, and don’t repeat the same folder in both. For example, for blobs at inbound/eml/, set Container Path to inbound/ and Directory to eml/ so the resolved path is inbound/eml/, not inbound/inbound/. Open the connection and check its Container Path before you set this field.
- Wildcard file name patterns are case-sensitive. *.eml and *.EML are different patterns. Use the casing your files actually use.
To add more directories, click More Files. You can include up to 5 directories.
Click Next.
Add an Object Name and an Object API Name for the UDLO. See Data Lake Object Naming Standards.
- Data Cloud appends __dll to the API name of the underlying Data Lake Object. For example, an Object API Name of Email_Archive_264_UDLO is queryable as Email_Archive_264_UDLO__dll. Account for this suffix in any downstream queries, tooling, or code references.
From the Data Space dropdown list, select a data space in which to create the new UDMO or a data space from which to select an existing UDMO.
Map the UDLO to a UDMO.
- To create a new UDMO, click New.
- To use an existing UDMO, click Existing, and select a UDMO from the list.
Optionally, leave the checkbox selected to create a search index configuration for the UDMO using system defaults that automatically selects text fields and a chunking strategy for each field. You can deselect the checkbox and create a search index configuration later if you choose not to do so now.
Click Next, or if you created a search index configuration, review the details, and save your work.
After establishing the connection from your external blob store, set up a file notification pipeline to notify Data Cloud whenever files are added, updated, or deleted from your external blob store.

Send File Notifications from Azure Storage to Data 360

Create a file notification pipeline for Azure Storage to notify a Salesforce connected app whenever unstructured data files are added, updated, or deleted from a bucket.

Required User Permissions

User Permissions Needed
Azure storage account and container	Contributor role set at the resource group level
Azure Key Vault Crypto Officer, Key Vault Secrets Officer, and Key Vault Data Access Administrator	Set at the Resource Group level
Azure CLI	-
Azure Functions Core Tools	-

Create a Private-Public Key Pair and Certificate

To configure OAuth for the connected app you will use in your file notification pipeline, create a private/public RSA key pair and a digital x509 certificate.

If you already have a connected app that you want to use, ensure that you have the private/public RSA key pair you used to create the x509 certificate for that app, as you need them to enable OAuth in a subsequent step.

Note

From your terminal, change directories to any folder.
Create the private/public key pair.
1openssl genrsa -out keypair.pem 2048

Create a digital certificate from the key pair.

1openssl req -new -x509 -nodes -sha256 -days 365 -key keypair.pem -out certificate.crt

Complete the questions as prompted.

Create a pkcs8 private key from the key pair.

1openssl pkcs8 -topk8 -nocrypt -in keypair.pem -out private.key

Keep the private/public key pair and the digital x509 certificate, as you need them in the following task.

Download the Azure Storage File Notification Installer Script and Function

Download the Azure Storage file notification installer script.
Download and unzip azure_function.zip.

Set Up A Connected App and OAuth Settings

Set up a connected app to use in the file notification pipeline and apply necessary OAuth settings.

Create a basic connected app. If you already have a connected app, continue to Enable OAuth settings for the API integration.
Enable OAuth settings for the API integration.
1. To use JWT OAuth Flow, select Use Digital Signatures.
2. Enter the callback URL (endpoint) that Salesforce calls back to your application during OAuth. It’s the same as the OAuth redirect URI.
3. Click Choose File, and select the certificate (.crt file) you created in Create a Private/Public Key Pair and Certificate.
4. Set these OAuth scopes: Manage user data via APIs (api), Perform requests at any time (refresh_token, offline_access), and Manage Data Cloud Ingestion API data (cdp_ingest_api).
5. Make sure Issue JSON Web Token (JWT)-based access tokens for named users is unchecked. If this option is checked, the org issues JWT-format access tokens that the Data Cloud token-exchange endpoint (/services/a360/token) rejects, and the file notification pipeline can’t acquire a Data Cloud token.
6. Note the Consumer Key and Callback URL.
Click Save.
In Setup, search for OAuth and OpenID Connect Settings.
Turn on Allow OAuth Username-Password Flows.

Using the Consumer Key and Callback URL created in Enable OAuth settings for the API integration. Paste the following URL into your browser using the noted values.

1<YOUR_ORG_URL>/services/oauth2/authorize?response_type=code&client_id=<YOUR_CONSUMER_KEY>&scope=api refresh_token cdp_ingest_api&redirect_uri=<YOUR_CALLBACK_URL>&code_challenge=SHA256

When prompted to provide permission for each of the scopes you requested, click Allow.
(Optional): If you want to verify that the app is correctly authorized, use the Quick Find search for “Connected Apps OAuth Usage”. Your connected app should be listed.

Depending on your downstream Data 360 usage (for example, querying or profiling ingested data), you may also need the cdp_query_api and cdp_profile_api scopes on the connected app.

Note

Run the Install Script

Separate installer scripts are provided for macOS/Linux and for Windows.

macOS and Linux

In the directory where you extracted the file notification installer script, go to the installers/azure folder.
Open input_parameters_azure.conf and replace the environment variable values with your values.

Run the installer script. This command makes the file notification script executable.

1chmod +x setup_azure_file_notification.sh
2 ./setup_azure_file_notification.sh input_parameters_azure.conf

macOS users: When you run the installer script, you are transferred to the UI to authenticate. After authentication, return to the terminal. The script continues to run in the terminal.
Linux users: When you run the installer script, an authentication link and code is displayed in the terminal. Copy the link and code to authenticate from your browser.

Windows

The Windows installer uses a PowerShell script and a .txt configuration file instead of the .sh script and .conf file used on macOS and Linux.

In the directory where you extracted the file notification installer script, go to the windows_installers/azure folder.
Open input_parameters_azure_windows.txt and replace the environment variable values with your values. This file uses the same variables as input_parameters_azure.conf; see the Azure Storage Variable Reference.
In Windows PowerShell, run the installer script, passing the configuration file with the -ConfigFile parameter.
1.\setup_azure_file_notification_windows.ps1 -ConfigFile input_parameters_azure_windows.txt

When you run the installer script, a browser window opens and redirects you to the Azure portal to authenticate. Log in and close the window. The script continues to run in PowerShell to completion.

Azure Storage Variable Reference

Refer to these variables when creating a file notification pipeline from an Azure blob store to Data 360.

Variable	Definition
`RESOURCE_GROUP`	Contains the name of your Azure resource group
`LOCATION`	Your Azure location
`CONTAINER_NAME`	The name of the container with your unstructured data files. Container name must start and end with a letter or number, and can only contain lowercase letters, numbers, and hyphens
`STORAGE_ACCOUNT`	Contains the ID of your Azure storage account
`SF_USERNAME`	Your Salesforce org username
`SF_LOGIN_URL`	Your Salesforce org login URL
`SOURCE_CODE_LOCAL_PATH`	The local path for your function’s source code
`APP_NAME`	The name of the function app
`TOPIC_NAME`	The name of the system topic used to publish events
`SUBSCRIPTION_NAME`	The name of the topic subscriber that listens to the system topic
`KEY_VAULT_NAME`	Contains the name of your Azure Key Vault instance
`CALLBACK_URL`	Contains the callback URL (endpoint) that Salesforce calls back to your application during OAuth
`PEM_FILE_PATH`	Contains the complete path to the keypair.pem file on your local machine. For example: `/Users/Name/Documents/keypair.pem`
`CONSUMER_KEY_VALUE`	The Consumer Key created in your connected app

Troubleshooting File Notifications

These common issues can occur when setting up a connected app or Azure Storage file notification in Data 360. If you don’t see data connected in your org after loading unstructured data into Azure and creating a file notification pipeline, review these requirements.

Connected App

Ensure that you have:

Used a valid consumer key and RSA private key. If you’re using a consumer key from a previously configured connected app, check that the connected app and Salesforce org still exists.
Set all three required scopes in the connected app, as documented.
Ensured that Issue JSON Web Token (JWT)-based access tokens for named users is unchecked. When this option is checked, the pipeline receives a JWT-format access token that Data Cloud’s token-exchange endpoint rejects.
Authenticated your connected app. In the Quick Find, search for “Connected Apps OAuth Usage”. You should see your connected app and a user count of 1.
Uploaded a .crt file in the connected app, not a .pem or .key file.
Used a .pem file when creating the RSA private key.
Used a single set of keys (.crt, .pem, and .key) for both the cloud function and the connected app.
Used the correct pattern for key names, as specified in the variable reference.

Unstructured Data Files and UDLO Creation

Ensure that you have:

Aligned the directory structure of the external blob store source directory and the UDLO directory. Data 360 looks for files at <Container Path>/<UDLO Directory>. For example, if the connection’s Container Path is data/ and you configure the UDLO Directory field as files, Data 360 looks for files at <Cloud Provider Bucket>/data/files. Don’t repeat the same folder in both fields, as that produces a doubled path that matches no files.
Correctly set the file type extensions and have matched the case used in the file names for the unstructured data files when creating the UDLO.

Installer Script and Cloud Function

Ensure that you have:

Properly set all input parameters.
Provided a valid Salesforce Org login url and username.
Uploaded the correct, latest cloud function zip for your notification pipeline. For example, don’t use the GCS cloud function zip for an Azure notification pipeline.
Set up access to the Azure portal from your Salesforce account.

Additional Troubleshooting

Section	Issue	Reason	Solution
Connected App	400 error in cloud function logs	The certificate (`.crt`) file used in the connected app is not associated with the RSA private key file (`.pem`) uploaded to the cloud provider.	Recreate the keys and replace the existing key (`.pem`) file uploaded to the cloud provider with the certificate file used in the connected app.
Connected App	403 error in cloud function logs	The connected app Org URL does not properly execute or is not available.	Ensure OAuth scopes are set properly and that the Org URL is reachable from a browser.
Connected App	400 error in cloud function logs	The connected app policy settings are incorrect.	Edit the policies in the connected app: Set IP configuration to Relax IP restrictions and enable OAuth username and password.
Data explorer	Files uploaded to the cloud bucket/container are not refreshed in UDLO/UDMO	Incorrect file types were uploaded, for example, a UDLO is set to accept PDF files but HTML files were uploaded.	Upload the correct file types, or create new UDLOs for the file types and upload the files again.
Azure Installer	Storage Account can’t be upper case and should be between 3 to 24 characters in length	Storage Account uses uppercase characters or the character count is 2 or above 24.	Ensure the Storage Account uses lowercase characters and a character count between 3 to 24.
Azure Installer	Not able to generate CDP access token	The name or pattern of the secret keys does not match `CONSUMER_KEY_<YOUR_OWN_SUFFIX>` and `RSA_PRIVATE_KEY_<YOUR_OWN_SUFFIX>`.	Ensure your secret keys use this pattern: `CONSUMER_KEY_<YOUR_OWN_SUFFIX>` and `RSA_PRIVATE_KEY_<YOUR_OWN_SUFFIX>`.

Create an Unstructured Data Connection from Azure Storage

Create a UDLO for Unstructured Data from Azure Storage

Send File Notifications from Azure Storage to Data 360

Create a Private-Public Key Pair and Certificate

Download the Azure Storage File Notification Installer Script and Function

Set Up A Connected App and OAuth Settings

Run the Install Script

macOS and Linux

Windows

Azure Storage Variable Reference

Troubleshooting File Notifications

See Also