Write and Validate Custom Scripts

Write custom scripts for batch data transforms in a standard Python environment by using your preferred IDE and validate them locally against a sandbox.

Edition Table
Available in: Developer, Enterprise, Performance, and Unlimited Editions. See Data 360 edition availability.
Permission Sets Needed
To write custom scripts and validate them locally against a sandbox:Permission set:
  • Data Cloud Architect

  1. Set up your Data 360 environment.

    1. Verify that you have access to a Data 360 sandbox with appropriate permission sets. If you don’t have a sandbox, create one. See Create a Data 360 Sandbox.

    2. Enable Code Extension in Data Cloud Setup using the Feature Manager. See Enable Data 360 Features in Feature Manager.

    3. Verify that you ingested data into your sandbox to create the DLOs or DMOs that your script references.

  2. Set up Salesforce CLI for code extension and create a script package. For more information, see Set Up Salesforce CLI for Code Extension.

  • Work exclusively with either DLOs or DMOs in a single script.
  • Add any additional Python dependencies to requirements.txt.
  • Use only pip-installable dependencies. Libraries that require OS-level installations or system configurations don’t work in Data 360.
  • Use clear function names and comments, and handle errors gracefully.
  • Test your logic frequently in your selected environment.
  • Use logging for debugging and monitoring.

Column names and object API names use different casing in your script.

Reference column names in lowercase, even though each field keeps its original casing in the Data 360 UI and in the field’s API name. For example, use unitprice__c for the UnitPrice__c field. A mixed-case column name returns a column does not exist error.

Use lowercase column names in every PySpark expression that references a column, such as col(), withColumn(), select(), and drop().

When you test locally, the script returns the same column names that Data 360 returns. Columns that resolve during local testing also resolve after you deploy the script.

Incorrect:

Correct:

Pass DLO and DMO API names exactly as they appear in Data 360, such as Account_Home__dll.

Example:

In code extension, you can’t use a DLO or DMO that one batch data transform writes to as an input object for another batch data transform.

Use one of these write modes when writing to DLOs or DMOs in your script.

Write ModeDescriptionExample
APPENDAdds new records to the target object without affecting existing data.client.write_to_dlo("Target_DLO__dll", df, write_mode=WriteMode.APPEND)
OVERWRITEReplaces all existing data in the target object with new data.client.write_to_dlo("Target_DLO__dll", df, write_mode=WriteMode.OVERWRITE)
MERGEUpdates existing records and adds new records based on key fields.client.write_to_dlo("Target_DLO__dll", df, write_mode=WriteMode.MERGE)
MERGE_UPSERT_DELETEUpdates, adds, or deletes each record based on the _merge_record_type column that you add to your DataFrame.client.write_to_dlo("Target_DLO__dll", df, write_mode=WriteMode.MERGE_UPSERT_DELETE)

Add the _merge_record_type column to your DataFrame and set it to UPSERT for records to update or add, or DELETE for records to remove. withColumn() requires a column expression rather than a plain string, set the value with lit().

A single write operation can contain both upserts and deletes. Data 360 matches each record to the target object by primary key. A record set to UPSERT updates the matching record, or adds the record if no match exists. A record set to DELETE removes the matching record; If no match exists, the delete operation does nothing.

Your DataFrame must meet these requirements:

  • Include the _merge_record_type column, or the write fails with an InvalidDatasetForMergeUpsertDelete error.
  • Set _merge_record_type to UPSERT or DELETE on every record. Data 360 skips records that are set to any other value.

Example

Use the same pattern with client.read_dmo() and client.write_to_dmo() when your script uses DMOs.

  1. Go to your code extension package folder.

  2. Create and activate a Python virtual environment in the package folder.

    macOS and Linux

    Windows (PowerShell)

    Windows (Command Prompt)

  3. Add required dependencies to requirements.txt.

  4. Install the dependencies.

    macOS and Linux

    Windows

To add new dependencies during development, always add them to requirements.txt and then install by using this command. This process ensures that all dependencies are included in your deployment package.

  1. Write your script in payload/entrypoint.py to reference the ingested data.

  2. Test your script.

    • --entrypoint <path_to_entrypoint> (required): Path to your script entry point (typically ./payload/entrypoint.py in the default package layout).
    • --target-org <org_alias> (required): The alias against which you’re testing your script.

    Example:

  3. Optionally, control the number of rows returned during local testing.

    By default, Data 360 queries return 1,000 rows when you test locally. To validate against a larger dataset, raise the row limit. To iterate faster on a smaller sample, lower it. Set the row limit in entrypoint.py before your script reads any data.

    1. Import config in your entrypoint.py.

    2. Set the row limit to the number of rows you need. For example, to return 5,000 rows:

      To remove the limit entirely and fetch all records, set the value to None:

    If you package and upload your entrypoint with this setting, the package includes the row limit, which then applies when run in Data 360. Remove or reset the row limit before packaging for deployment.

  4. Update the Data 360 configuration file after changing your script.

    The sf data-code-extension script scan command doesn’t work as intended for DMO-to-DMO transforms. You must manually craft the config.json file. See Configure config.json for DMO-to-DMO Transforms.

Deploy a Custom Script to Data 360 Sandbox by Using UI.