Write and Validate Custom Scripts (Beta)

Write and validate custom scripts for batch data transforms by using Data Custom Code SDK. This section describes all supported authoring environments and best practices.

Code extension is a pilot or beta service that is subject to the Beta Services Terms at Agreements - Salesforce.com or a written Unified Beta Agreement if executed by Customer, and applicable terms in the Product Terms Directory. Use of this pilot or beta service is at the Customer's sole discretion.

Edition Table
Available in: Developer, Enterprise, Performance, and Unlimited Editions. See Data 360 edition availability.

Permission Sets Needed
To write and validate custom script for batch data transforms:	Permission set: Data Cloud Architect

Set up your Data 360 environment.
1. Make sure that you have access to a Data 360 sandbox with appropriate permission sets. If you don’t have a sandbox, create one. See Create a Data 360 Sandbox.
2. Make sure that you ingested data into your sandbox to create the DLOs or DMOs that your script references.
Install the SDK and create a project. See Set Up Data Custom Code SDK.

Work exclusively with either DLOs or DMOs in a single script.
Add any additional Python dependencies to requirements.txt.
Use only pip-installable dependencies. Libraries that require OS-level installations or system configurations don’t work in Data 360.
Use clear function names and comments, and handle errors gracefully.
Test your logic frequently in your chosen environment.
Use logging for debugging and monitoring.

Use exact API names for Data 360 object fields. Field names follow the FieldName__c pattern.

Use one of these write modes when writing to DLOs or DMOs in your script.

Write Mode	Description	Example
APPEND	Adds new records to the target object without affecting existing data.	`client.write_to_dlo("Target_DLO__dll", df, write_mode=WriteMode.APPEND)`
OVERWRITE	Replaces all existing data in the target object with new data.	`client.write_to_dlo("Target_DLO__dll", df, write_mode=WriteMode.OVERWRITE)`
MERGE	Updates existing records and adds new records based on key fields.	`client.write_to_dlo("Target_DLO__dll", df, write_mode=WriteMode.MERGE)`

Choose your development environment based on your workflow and preferences.

Environment	Use Case	File Type
Standard Python Virtual Environment	Quick prototyping and simple scripts	`payload/entrypoint.py`
JupyterLab	Data exploration and iterative development with interactive notebooks (local only)	`account.ipynb`
Visual Studio Code Dev Containers	Production-ready development and team collaboration. Containerized environment that closely matches Data 360's production compute environment.	`payload/entrypoint.py`

Go to your SDK project folder.
Create and activate a Python virtual environment in the SDK project folder.
Add required dependencies to requirements.txt.
Install the dependencies.

To add new dependencies during development, always add them to requirements.txt and then install by using this command. This process ensures that all dependencies are included in your deployment package.

Write your script in payload/entrypoint.py to reference the ingested data.
Test your script.
Update the Data 360 configuration file after changes to your script.

The datacustomcode scan command doesn't work as intended for DMO-to-DMO transforms. You must manually craft the config.json file. See Configure config.json for DMO-to-DMO Transforms.

After developing your logic in the notebook, copy and refactor it to entrypoint.py for deployment to Data 360.

Go to your SDK project folder.
Start JupyterLab.

Wait for the container to start. The terminal output displays a URL (http://localhost:8888).
Open the displayed URL in your web browser to access JupyterLab.
JupyterLab runs in an isolated environment with its own temporary storage, so rerun the configuration command each time you start JupyterLab.
Write and test your custom script in the notebook.
1. Double-click the account.ipynb file to open the notebook.
2. Review the existing code cells in the notebook that provide the base logic for batch data transformation. Write your transformation logic directly in the notebook cells.
3. Use Shift+Enter to execute each cell and test your logic incrementally.
4. Add new cells to experiment with different approaches, and validate your logic step by step.
5. If you need additional packages, add them to requirements.txt and restart JupyterLab to ensure they're available.
6. Save your notebook with the final transform logic.
Copy your notebook logic to entrypoint.py.
Test your script locally.
Update your Data 360 configuration.

The datacustomcode scan command doesn't work as intended for DMO-to-DMO transforms. You must manually craft the config.json file. See Configure config.json for DMO-to-DMO Transforms.
Stop JupyterLab.

Install VS Code.
Install Microsoft’s Dev Containers extension.
Open your SDK project folder in VS Code so that the .devcontainer folder is at the root of the File Explorer.
Start the Dev Container.
1. Open the Command Palette (on Mac: Cmd + Shift + P; on Linux: Ctrl + Shift + P).
2. Select Dev Containers: Rebuild and Reopen in Container, and then wait for the Docker image to be built. (This process takes a few minutes the first time.)
Dev Containers get their own temporary file storage, so rerun the configuration.
Develop your script.
1. Write your logic in payload/entrypoint.py.
2. Use the integrated terminal, debugger, and other VS Code features as usual.
3. If you need additional packages, add them to requirements.txt and restart the Dev Container to ensure they're available.
To test your script, open a terminal within the Dev Container window and run your script.
Update the configuration after making your changes.

The datacustomcode scan command doesn't work as intended for DMO-to-DMO transforms. You must manually craft the config.json file. See Configure config.json for DMO-to-DMO Transforms.
To shut down the container, close VS Code or use the Command Palette to run Dev Containers: Close Remote Connection.

Deploy a Custom Script to Data 360 Sandbox by Using UI.

Considerations When Writing Custom Code (Beta)