The Salesforce CLI has made, and will continue to make, some breaking changes to how deploy and retrieve hooks work. If you create or use custom plugins that use predeploy, postdeploy, preretrieve, postretrieve, or postsourceupdate hooks, there are changes to both the payload and behavior of those hooks that could affect you.

In this blog post, we’ll explain why we’re making these changes. We’ll also walk through the details of the changes and provide examples of what will and won’t break, along with potential workarounds and some forward looking statements.

Hooks background and example use case

Let’s take a look at how hooks used to work. Before our recent changes, deploy and retrieve hooks provided a way to intercept metadata (code, configuration, etc) that was going from your local machine to a Salesforce org (or vice versa) and made changes to the deployment/retrieve operation.

For instance, one real-world hook we found replaced placeholders in local source with secrets. To understand how that worked, let’s consider the use case of a Salesforce Connect external data source. Let’s say that you want to deploy and run tests with an external data source, but you don’t want to store a password field in source control.

Your hook could replace $xdsPassword

with a value from the environment

before deployment.

Then, a developer’s machine or a CI system that has the environment variable can deploy the ExternalDataSource metadata without ever writing the password to the source files.

How hooks used to work during a deploy

Until CLI version 7.111, a force:source:deploy command worked like this:

1. On a deploy, the CLI would convert all of your source-formatted code to metadata API format (reading each file and writing it to a temporary directory).
2. The predeploy hooks would fire, and the hook payload would contain a mdapiFilePath which would point to each temporary file along with metadata about that file
3. Hooks code would receive that array of files and perform any logic
4. In our example, the custom plug-in would look for ExternalDataSource types, and then open that file and modify password value with the value from the environment
5. The hook would exit, and the CLI would continue by zipping the (now modified) metadata and sending it to the metadata API

As a plug-in developer, you could ensure consistent execution of your hook code on every deploy done from either the CLI or the VSCode extensions since those were calling the CLI.

force:source: retrieve and force:source:pull worked nearly the same, but in reverse. They would retrieve the zip file, unzip to a temporary directory, fire postretrieve hooks to let you modify the result, and then convert that and write to your source folders.

Meet the new source deploy/retrieve library

We heard from our users that they wanted faster deployments, and that large deployments were particularly slow. Additionally, VSCode extension users who changed one file wanted that to deploy to their Salesforce org as quickly as possible. To address these concerns, and increase the speed of deploy and retrieve, we created the Source Deploy/Retrieve library (SDR).

The SDR speeds up deploy/retrieve by:

Reducing file system operations. Rather than write each converted file to the temporary directory, write that folder to a zip file, and upload the zip, the SDR streams files from the source code to an in-memory zip file that goes directly to the metadata API. The only filesystem operation is reading the files that will be deployed. Filesystem operations are slow, especially on Windows machines, and avoiding them means faster deploys. Large projects might have 10,000 metadata files in a deployment (and potentially many more once all those decomposed children of CustomObject become individual files!)

Since not all of you were using a recent version of the CLI that includes the SDR, we looked at the metrics for October and saw a 36% reduction in median deploy time and a 57% reduction in retrieve time (and a whopping 75% reduction on deployments at the 90th percentile…ones that were taking pretty long).

Allowing direct use by VSCode. The Salesforce VSCode Extensions originally used the CLI to deploy and retrieve metadata. Since the CLI was loaded as a child process, its usage added a lot of overhead time to the deploy/retrieve commands in VS Code. Calling the CLI directly also meant that we were automatically parsing input flags, doing validations, loading help messages, etc., most of which is not needed each time you “deploy on save,” for example. The Salesforce Extensions now make calls directly to the SDR and save a lot of time not spawning child processes or doing unnecessary CLI-related tasks.

In addition to these performance gains, the SDR is an open-source library that can be used by anyone else writing metadata tools — it’s non-trivial to maintain conversion logic for the hundreds of metadata types and the dozens of new types that appear in each Salesforce release. If your custom plug-in deals with metadata or the metadata API, this is a major improvement for you.

To achieve these gains, we’re adopting the SDR. However, that breaks some use cases for hooks.

What’s using SDR

As of version 7.135.0, the CLI currently uses the SDR for:

• source:deploy, source:retrieve, source:convert, and source:delete (and their report flavors)
• source:beta:pull, source:beta:push , and source:beta:status which deliver similar performance gains to those shown above
• mdapi:beta commands: deploy , retrieve, and deploy:report

In the Salesforce VS Code Extensions, we’re using the SDR across the following areas:

• Context menu & Command palette commands: Deploy/Retrieve Source from Manifest, Deploy/Retrieve Source from Org
• Org browser
• Conflict detection
• Apex test functionality

How the SDR breaks hooks

First of all, postsourceupdate was removed for retrieve commands in 7.112.0, so hooks that rely on it will not run. Source conversion happens in the stream as soon as the retrieved zip is downloaded and unzipped, so postretrieve now happens at the same stage that postsourceupdate did (that is, after local source format files are written).

Secondly, there’s no temporary directory path for you to modify code. If your hooks were expecting to use the temporary location of metadata-formatted source, it doesn’t exist anymore.

Thirdly, VSCode extensions now call the SDR directly (instead of the CLI) for deploy/retrieve. Hooks won’t fire at all for deploy/retrieve operations initiated from VSCode unless you’re using the CLI in the VSCode terminal.

Examples and workarounds

In our example from above (replacing a placeholder with secrets from the environment), the plug-in can see what’s going to deploy from the predeploy hook. But without the temporary directory, it cannot make any changes to the source code before it deploys. In this case, the developer could do one of the following:

1. Write a custom deploy or push command that includes the secrets step. It’s significantly easier now that our commands are open source and the SDR is available.
2. Write a shell script that effectively does what the original hooks process did:
   1. Convert source files to a temporary metadata folder (force:source:convert)
   2. Look through that folder and modify anything you like
   3. Deploy that temporary metadata folder (force:mdapi:deploy)
   4. Delete the temporary metadata folder
3. Use a postdeploy hook to recreate the original process as a small second deployment:
   1. See if any ExternalDataSource files were deployed and whether they contain the placeholder
   2. Convert those files to a temporary metadata folder
   3. Wwap the placeholder/secret
   4. Deploy that temporary metadata folder
   5. Delete that temporary metadata folder

Another hook that we saw strips out the UserPermissions property from Profile and PermissionSet during a retrieve. If your plug-in was previously using a postretrieve hook, you’ve already noticed that force:source:retrieve has a different hook payload. It still contains the path to the retrieved files, which your plugin could use to look for any Profile or PermissionSet and rewrite those files without UserPermissions. In this case, the developer could:

1. Rewrite the hook code to handle both the new and old hook payloads
2. Once all commands are using SDR, remove the handler for the pre-SDR hook format

Another plug-in we found uses hooks for destructive changes on a deploy — a feature we recently added to the stock source:deploy command. There’s no need to continue using the plug-in for destructive changes.

One more: a developer noticed a bug in Salesforce’s source tracking. After renaming a field, the CustomField would appear in changes, but a report that used the field would not (even though its source code changed to reflect the field’s change). The plug-in used a preretrieve hook to examine local source code files and insert any report that references the field into the package.xml, so that the report would be included in the retrieve. In this case, the developer could:

1. See the retrieved files (via a postretrieve hook)
2. Have their hook kick off another retrieve (force:source:retrieve -m Report:SomeReport) to get the missing report into local source

Heads-up: push and pull will break next

We rewrote push and pull to use the SDR, and those changes are currently in beta starting in 7.25.0. We’ll eventually flip the new version to be the default, which will cause the hooks to change as discussed above.

Heads-up: VSCode isn’t guaranteed to use the CLI

As part of the move to the SDR, Salesforce Extensions have shifted to a “library strategy” and no longer call the CLI directly across our Context menu and Command palette commands. As of the latest version of Salesforce Extensions (53.1.0), the CLI is only used behind the scenes for some base debugger setup and auth functionality. Any hooks plugins (even in the new style) will not fire the same hooks in VS Code moving forward.

Transitional workarounds

The existing push and pull will move under the force:source:legacy subtopic, so you can continue to use them as they currently work even after the new versions become the default.

You can use older versions of the CLI (we guarantee at least 40 previous versions will be available). Versions before 7.112 didn’t include the SDR for deploy/retrieve and use the previous versions of hooks.

Similarly, you can use older versions of Salesforce Extensions — around 200 previous versions can be installed directly from VS Code. The SDR was introduced to VS Code in version 48.5.0, but the ability to switch back to the CLI implementation and use hooks was available through version 53.1.0. To switch back to the CLI implementation from an earlier version, search for Experimental: Deploy Retrieve in the VS Code settings.

Note: We really mean these as transitional workarounds. Besides missing out on the SDR performance gains, you’re also opening yourself to all the bugs and security updates that have been fixed in newer versions of the CLI and Salesforce Extensions for VS Code.

Conclusion

Breaking changes are always hard. We hope that this post has helped you to understand why we made these recent changes, what might be impacted, and what you can do to address them. To keep up with CLI updates, such as the upcoming change to the push and pull commands, follow the SFDX release notes or run sfdx whatsnew. If you’re creating custom plug-ins, please explore the SDR and Salesforce’s source and mdapi commands that implement it.

About the author

Shane McLaughlin is a Developer on the Salesforce CLI team and has been building on the Salesforce Platform since 2011. He’s @mshanemc on both Twitter and GitHub.