Creating Custom Job Steps
If there is no system step that does what you need to do, a developer must develop a custom job step. The developer first writes a task-oriented or chunk-oriented CommonJS script module or a legacy pipeline to describe what the step does. The developer includes the module or pipeline in a cartridge along with a JSON or XML file that describes the step, and then uploads the cartridge to the site. The custom steps are then available to the administrator creating jobs. The topics in this section are.
You can define what a custom step does using a task-oriented CommonJS script module that exposes a function to be called as the main function for the job step. When administrators create jobs using Business Manager, they set parameters that are available as scriptable objects for the module's function and for the
dw.job.JobStepExecution object. The
dw.job.JobStepExecution object allows read-only access to information about the current step execution and job execution.
To control the exit status, the script module's function can return a
dw.system.Status object. If the script finishes with an unhandled exception, the exit status code is ERROR, and the error status flag is true by default. If no status object is returned and no exception occurs, the status code is OK by default.
The following example is a task-oriented CommonJS module.
You can define what a custom step does using a chunk-oriented CommonJS script module that reads and processes items in chunks of size S. If the list contains more items than can be processed in one chunk, a new chunk is started. A chunk-oriented script can include any series of processing steps, not just database transactions. A job step that uses a chunk-oriented script allows fine-grained progress monitoring because the number of elements that are written is updated each time a chunk is finished.
Chunk Processing Example
In this example, a list of 15 orders is exported to a file. Each chunk includes four orders.
- Read Order 1.
- Process Order 1.
- Read Order 2.
- Process Order 2.
- Read Order 3.
- Process Order 3.
- Read Order 4.
- Process Order 4.
- Write Order 1.
- Write Order 2.
- Write Order 3.
- Write Order 4.
- Read Order 5.
- Process Order 5.
- Read Order 6.
- Process Order 6.
- Read Order 7.
- Process Order 7.
- Read Order 8.
- Process Order 8.
- Write Order 5.
- Write Order 6.
- Write Order 7.
- Write Order 8.
- Read Order 9.
- Process Order 9.
- Read Order 10.
- Process Order 10.
- Read Order 11.
- Process Order 11.
- Read Order 12.
- Process Order 12.
- Write Order 9.
- Write Order 10.
- Write Order 11.
- Write Order 12.
- Read Order 13.
- Process Order 13.
- Read Order 14.
- Process Order 14.
- Read Order 15.
- Process Order 15.
- Write Order 13.
- Write Order 14.
- Write Order 15.
This chunk read, processed, and wrote fewer items than the previous chunks, because only three items were left on the list.
|Function||Required or Optional||Purpose|
|read-function||Required||Returns one item or nothing if there are no more items.|
|process-function||Required||Transforms items and applies business logic to them. It receives the item returned by the read function, performs a process, and returns one item.The item returned can be the same item that the read function returned if no processing logic is necessary, or it can be a new item of a different type. If the process function returns nothing, then the read function item is filtered and doesn't appear in the list of items to be written later.|
|write-function||Required||Receives a list of items. The list size matches the chunk size or, if the number of items in the last available chunk is smaller, it is smaller. The write function returns nothing.|
|total-count-function||Optional||Returns the total number of items that are available. Called by the framework exactly once before chunk processing begins. A known total count allows better monitoring, for example, to show that 50 of 100 items have already been processed.|
|before-step-function||Optional||Executed before a chunk step begins. Implements logic before all items of all chunks are read, processed, and written.|
|before-chunk-function||Optional||Executed before a chunk begins. Implements logic before a chunk of S items is read, processed, and written.|
|after-chunk-function||Optional||Executed after a chunk finishes. Implements logic after a chunk of S items has been read, processed, and written successfully.|
|after-step-function||Optional||Executed after a chunk step finished successfully. Implements logic after all items of all chunks are read, processed, and written successfully.|
When administrators create jobs using Business Manager, they set parameters that are available as scriptable objects for each exposed module function and for the
dw.job.JobStepExecution object. The
dw.job.JobStepExecutionobject allows read-only access to information about the current step execution and job execution. You can't define the exit status for a chunk-oriented script module. Chunk modules always finish with either OK or ERROR.
The following example shows a chunk-oriented CommonJS module.
In a task-oriented script module or pipeline for a custom job step, you can create an object of type
dw.system.Status to control the exit status and error message for the step. The
Status object includes a boolean flag that indicates whether or not the status is error, and also a status code and message. Messages appear in Business Manager and are written to log files. You can't define the exit status for a chunk-oriented script module. Chunk modules always finish with either OK or ERROR.
The job framework uses the following rules to detect the step status and determine if job execution can continue.
|dw.system.Status.isError()||dw.system.Status.getCode()||Status Code||Error Status Flag||Job Execution Behaviour (if not handled by a transition)||Notes|
|False||<CUSTOM>||<CUSTOM>||False||Continue||Custom status codes can't contain any comma (,) or wildcard (*) characters, include leading or trailing whitespace, or exceed 100 characters.|
|True||<CUSTOM>||ERROR||True||Stop||Custom status codes for statuses that represent an error are not supported. If a custom status code is used the code is replaced with ERROR.|
If a pipeline or script module does not return a
dw.system.Status object or if a pipeline or script module's function fails before a
Status object is returned, the following rules apply.
|Pipeline||Status Code||Error Status Flag||Job Execution Behaviour (if not handled by a transition)||Notes|
|Finished with stop node||ERROR||True||Stop|
|Finished with interaction node||ERROR||True||Stop|
|Finished with unhandled exception||ERROR||True||Stop|
|Finished with end node||OK||False||Continue||The name of the end node isn't relevant|
|Stores an object that isn't a ||OK||False||Continue|
|Stores no object under key ||OK||False||Continue|
|Script Module Function||Status Code||Error Status Flag||Job Execution Behaviour (if not handled by a transition)||Notes|
|Runtime exceeds configured (or default) timeout||ERROR||True||Stop|
|Returns an object that isn't a dw.system.Status object||OK||False||Continue|
|Returns no object||OK||False||Continue|
steptypes.json file, which describes custom job steps, requires a specific JSON syntax. If a
steptypes.json file contains errors, the errors are logged and the steps are not registered. The system then loads steps from the
steptypes.json files of other cartridges. A typical error message for an invalid
steptypes.json file is
Invalid step [Step1]! Type with id [custom.MyCustomStep1] is unknown!.
To check the syntax of your
steptypes.json file, access the
steptypes.schema file here.
|Element||Contained By||Contains||Required or Optional||Notes|
|Required for ||Must not contain leading or trailing white space.|
|Required for ||Must not contain leading or trailing whitespace.|
|Optional||Must have value true or false. Default is true. If true, parameter is required. If false, parameter is optional.|
|Optional||Must have value true or false. Default is true. If true, steps of this type can be used for flows with organization scope. If false, steps of this type can't be used for flows with organization scope. |
|Optional||Must have value true or false. Default is true. If false, split flows that contain steps of this type are always executed sequentially and never in parallel. If true, split flows that contain steps of this type are executed in parallel, as long as: |
|Optional||Must have value true or false. Default is true. If true, steps of this type can be used for flows with one or more sites as scope. If false, steps of this type can't be used for flows with one or more sites as scope. |
|Optional||The type to which the parameter value is converted, either |
|Optional||Specifies whether leading and trailing whitespace for the parameter value entered in Business Manager is removed before the value is validated. Must not contain leading or trailing whitespace. Must have value true or false. Default is true.|
|Required||Indicates the data type of the parameter value. Must not contain leading or trailing whitespace. Must have value of |
|Required||Identifies the step. This is the name that users will see in Business Manager, so make it descriptive. Must begin with |
|Optional||References the function of the chunk-oriented script module to be executed after a chunk has finished. If not defined, no after-chunk logic is used.|
|Optional||References the function of the chunk-oriented script module to be executed after the step has finshed. If not defined, no after-step logic is used.|
|Optional||References the function of the chunk-oriented script module to be executed before a chunk begins. If not defined, no before-chunk logic is used.|
|Optional||References the function of the chunk-oriented script module to be executed before the step begins. If not defined, no before-step logic is used.|
|Required if no ||Defines a chunk-oriented step.|
|Required for ||Must be a numeric value. Must not contain leading or trailing white space.|
|Optional||Default when no value is entered for the parameter. Must have a valid data type value for the parameter (boolean, string, long, double, datetime-string, date-string, or time-string). If the parameter is assigned |
|Required||Must not contain leading or trailing whitespace or more than 256 characters.|
|Optional||Description of the step or status code. Not shown in Business Manager. Must not exceed 4000 characters.|
|Optional||Allowed values for the parameter. The value entered by the user in Business Manager is validated against this list. Must not contain leading or trailing white spaces.|
|Optional||The function of the script module to execute. Must not contain leading or trailing white space. If not defined, the script module must export a function named |
|Optional||Maximum length of a string value for this parameter. String entered in Business Manager is validated against this restriction. Only applicable if |
|Optional||Maximum numerical value for the parameter. Only applicable if |
|Optional||Minimum length of the string value for the parameter. Only applicable if |
|Optional||Minimum value for the parameter. Must not exceed the |
|Required||Path to the script module to be executed. Must not contain leading or trailing white space.|
|Required if ||A parameter for the step, which the user configures in Business Manager.|
|Required||Parameters for the step, which the user configures in Business Manager. Contains one or more |
|Optional||Regular expression that defines the allowed values for the parameter. Applicable only if |
|Required||Pipeline to execute for the step. Format must be |
|Required if no ||Defines a pipeline for the step.|
|Optional||Process function of the chunk script. If not defined, script must export a function named |
|Optional||Read function of the chunk script. If not defined, script must export a function named |
|Required if no ||Defines a task-oriented script for the step.|
|Required for ||Defines the error codes and descriptions for the step.|
|Optional||Defines the meta data for status codes returned for the step.|
|Optional||Must be an integer that sets the timeout in seconds for the script module's function. There is no default timeout, but setting a limit is recommended.|
|Optional||References the function to be executed to determine the total number of items. If not defined, no total count is used.|
|Optional||Indicates if the module requires a database transaction. Must have value true or false. Default is false. If this value is set to true, the job step executes as a single, potentially very large, transaction. To avoid a negative impact on system performance and allow more granular transaction control, keep the default setting of false. Implement transaction handling within the job step using the dw.system.Transaction API.|
|Required for ||Comma-separated list of values allowed for the parameter|
|Optional||References the write function of the chunk script. If not defined, script must export a function named |
The following table shows the results of possible settings of the
supports-site-context elements in the
|true||false||Job is executable only if other steps in the same flow can be executed for the entire organization.|
|false||true||Job is executable only if other steps in the same flow can be executed for one or more sites.|
steptypes.schema.json file to check the syntax of your
steptypes.schema file here.
To create a custom job step, you must develop a task-oriented or chunk-oriented CommonJS script module or legacy pipeline to define what the step does. You must also create a JSON or XML
steptypes file that describes the step, and then include the module or pipeline along with the
steptypes file in a cartridge uploaded to the site.
To create a custom job step, you must be familiar with creating and uploading a cartridge.
Create a cartridge.
Create a task-oriented or chunk-oriented CommonJS module to describe what the step does.
A best practice is to put the CommonJS module in the
cartridge/scriptsdirectory or subdirectory, for example,
Instead of using a CommonJS module, you can use the UX Studio plug-in to the Eclipse IDE to create a pipeline. However, we recommend using CommonJS modules for jobs in the new framework.
steptypes.xmlfile that describes the job step and put the file in the root directory of the cartridge.
You can describe more than one custom job step in the same file. Each cartridge can have only one
Upload the cartridge, and include it in the cartridge path.
For a production system, replicate the code to production, and activate the code version that includes your cartridge.
When an administrator creates a job in Business Manager, custom steps defined in the
steptypes.xml file are listed as steps available to add to the job.