The Salesforce CLI is a powerful command line utility. It decreases the complexity of working with Salesforce orgs for application development and delivery. Source changes between your local project and scratch orgs are tracked by the CLI. This feature is referred to as “Source Tracking“. In this blog post you will learn how source tracking works, in-depth. You can leverage this knowledge to troubleshoot the CLI issues in your day-to-day development cycle.

This is an advanced post and assumes that readers are familiar with building apps using Salesforce CLI. You can get hands-on experience with Salesforce CLI by completing the Trailhead Module here.

Source Tracking Overview

Let’s dive into how the source is tracked both in a project workspace and in a Salesforce org. Source tracking functionality can be better understood by breaking it down into the following topics:

• Source tracking on your local machine
• Source tracking on the Salesforce platform
• Conflict Resolution Mechanism

Conflict resolution helps manage conflicts between local files in a project and components in a scratch org.

Source tracking on your local machine

When you create an SFDX project, you get a scaffolded directory and a set of files. A few of those files facilitate the source tracking feature on your local machine. The most important being the .sfdx folder.

NOTE – You do not need to look into the .sfdx folder except for understanding this content. The implementation details of the .sfdx folder can change, including names of files and folders. Do not delete any folder or files in the .sfdx folder as they are necessary for the internal workings of the Salesforce CLI (except for log files). Also, do not build any custom sfdx plugins with the assumption that .sfdx folders and files will always exist.

An orgs folder gets created within the .sfdx folder the first time you execute source status/push/pull commands. A subfolder is also created under the orgs folder with a name equal to the username of your user that’s used for authentication.

In the screenshot below test-tdwuzfbnif4j@example.com is the folder generated under the .sfdx/orgs folder of the project source. Note the files under orgs/test-tdwuzfbnif4j@example.com that are outlined in red.

metadataTypeInfos.json contains all metadata types and their definitions. This information is used internally by the CLI. One usage of this file is to generate a package.xml file for the metadata deployment when a developer executes a force:source:push command.

sourcePathInfos.json contains a list of all directory paths, folder paths, and file paths as JSON objects. Each object holds the information listed below as object properties:

• Last time the file was changed/modified (changedtime,modifiedtime)
• The string representation of the file path (sourcePath)
• A hash value of the file (contentHash). A hash value is a unique value that corresponds to the content of the file.
• The size of the file(size)
• The state of the file indicating whether it is new or changed (state)
• A flag to indicate whether the path is a directory (isDirectory)
• A flag to indicate whether the path is an XML metadata file (isMetadataFile)
• A flag to indicate the path is a current workspace path(isWorkspace)

This information is used to make sure conflicts between files in an org and your project folder can be properly detected. The file is updated each time you execute source push and pull commands.

maxrevision.json stores the maximum revision value from the last valid metadata pull/push from a scratch org. More details on this are covered in the next section where we discuss SourceMember object.

Source tracking on the Salesforce platform

The Salesforce Tooling API exposes a SourceMember object that logs any changes to the org metadata and settings.

Let’s say you create an object inside your Salesforce scratch org, this change is tracked as a record in SourceMember object with a name of the object. The CLI provides error messages if the metadata data type is not supported for source tracking.

Note: We are working to get source tracking support for all metadata types. This is an ongoing effort.

The below command shows how to query SourceMember records using the CLI:

sfdx force:data:soql:query --query "SELECT ChangedBy, IsNameObsolete, MemberName, MemberType, RevisionCounter FROM SourceMember" --usetoolingapi

Somethings to note about SourceMember object records:

• Revision counter maintains the number of revisions that happened on the metadata file. The maximum value is stored in your local project in .sfdx/orgs/<username>/maxrevision.json.
• When metadata is deleted, IsNameObsolete becomes true. Note that entries in SourceMember are never deleted and instead SourceMember records are flagged to indicate their deletion.

Note: SourceMember object is continuously enhanced every release to improve source tracking capabilities. We don’t recommend building anything against this object.

Conflict Resolution Mechanism

To further understand how the Salesforce CLI manages conflicts, let’s explore how source status and source push commands work under the hood.

How force:source:status works

Source status command lists the metadata that has changed in local package directories and Salesforce orgs since the last time they were both in sync.

When a developer executes the source status command, the following steps are executed.

1. The Salesforce CLI collects files that have changed in the project workspace since the last sync.
2. The Salesforce CLI queries for SourceMember records with RevisionCounter field value greater than the max revision count stored in the maxrevision.json file.
3. If a SourceMember record is found in step 2 and a file exists in the local project for the component, the file is marked as a conflict.
4. The CLI output terminal displays local files that are changed/added, remote files that are changed/added, and files that have conflicts. Example output from the terminal is shown below.

Some key points to note:

• The CLI tracks local file and folder changes by comparing the current hash value of the file with the hash value of the file in sourcePathInfos.json and by inspecting the state property of  file path object in sourcePathInfos.json
• The above steps are simplified and details around authentication are skipped. For complete details, refer to the code in the sourceStatusAPI.ts file in the Salesforce-alm repo.

How force:source:push works

Sync the changes made in your workspace to your scratch org by pushing the changed source to it. To push the source use command sfdx force:source:push

When a developer executes the source push command, the following steps are executed.

• The CLI collects files that have changed in the project workspace since the last sync.
• The CLI queries SourceMember object records greater than max revision counter stored in maxrevision.json file.
• The CLI marks files as conflicts if a SourceMember object record is found for them.
• If conflicts are detected, the CLI informs the developer about these conflicts. The developer needs to resolve them and force push using the force flag.
• If no conflicts are detected, the CLI converts source to Metadata API format with package.xml and generate a zip artifact.
• The CLI deploys changes to the Salesforce org using the Metadata API.
• The CLI updates state and file contentHash in sourcePathInfos.json and maxrevision.json.

The above steps are shown in the flowchart below as well.

Some important things to note on the source push command:

• The CLI handles automatic generation of the package.xml manifest. It takes care of converting the source to metadata format.
• With Source format the metadata file is broken into smaller chunks (source decomposition). This helps versioning and is more human readable.
• Upon conversion from source format to metadata format, the CLI handles aggregating decomposed metadata. An example of this is the custom object folder in source format being broken into sub folders for fields, validation rules, list views record types and others. Source push composes the actual metadata by aggregating definitions from various subfolders. The command also generates a metadata zip file with the package.xml manifest.
• The source push command has a force flag (-f or --forceoverwrite) to force overwrite files in the Salesforce org. This is useful in case you are sure that what you have in your local project workspace should overwrite what exists in the Salesforce org.
• The above flow is simplified and details around authentication are skipped. For complete details refer to the sourcePushApi.ts file in the salesforce-alm repo.

Source pull command leverages similar logic as discussed above. For more information, explore the source code in the sourcePullApi.ts file.

Tips and Tricks

This section describes tips and tricks to help you troubleshoot issues with the Salesforce CLI in your day to day development cycle.

Identity and Resolve Conflicts

Salesforce CLI provides commands to help identify conflicting files. As seen in the conflict resolution section, source push and pull commands can show conflicting files and stop execution if conflicts exist.

To display conflicts, make sure to run the sfdx force:source:status command. This helps identify metadata files that conflict with metadata in scratch orgs. This ensures metadata that you have added/modified at the remote and local machine are displayed on the terminal.

To force push or pull, use force flag (-f or --forceoverwrite) to overwrite server files and local files, respectively.

If you are using a non-source-tracked org like a Developer Edition org, you can use the Source Diff (BETA) feature of Salesforce extensions for VS Code to find differences between the project workspace and the org.

.forceignore

To exclude source files from the sync between the workspace and the Salesforce org, use .forceignore. This file has similar functionality as .gitignore. You can indicate which source to exclude by specifying glob pattern matching. The details on where to put the .forceignore file and patterns are covered in Salesforce DX documentation.

If you think some metadata type is breaking your workflow as it’s not supported yet for source tracking, you can track them separately by keeping it in a separate folder and listing it in the .forceignore file.

For example, let’s imagine you are building a login flow since metadata for login flow is not source tracked. Create a folder named ‘unpackaged’ and keep the package.xml and metadata for login flows in the unpackaged folder. Then add the path of the unpackaged folder (force-app/main/unpackaged) to the .forceignore file. This way you can deploy the unpackaged folder using force:source:deploy to other orgs.

Troubleshoot Push/Pull Issues

• Errors are logged in the CLI output when source push/pull commands are executed.
• Often source push failures can be deployment failures. View error messages in your scratch orgs by navigating to Setup > Environments > Deploy > Deployment Status. These messages are also displayed in the command line output terminal. An example output is shown below.

Salesforce CLI Log Analysis

Salesforce CLI application code is written in TypeScript for Node.js runtime. During the execution of commands, if you encounter Node.js errors you can troubleshoot by inspecting logs.

Salesforce CLI allows you to set log levels for commands. The default level of log messages is warn. You can set the log level to one listed here. Log levels are in order of least to most information.

You can collect more log info by using –loglevel flag. As an example, execute sfdx force:source:push --loglevel=trace to collect logs while the source push command is executed.

If you are on macOS, then open the log file using the below command in the terminal.
open -a TextEdit $HOME/.sfdx/sfdx.log

If you are on a Windows machine then open the log file using the command below from your command prompt terminal or powershell.
start .sfdx/sfdx.log

If you are using Visual Studio Code editor and configured Visual Studio Code to launch from the command line, then type in the command below from Home directory.
code .sfdx/sfdx.log

When you are raising a ticket with support or creating an issue in the git repo of the project, we normally request trace logs. You can obtain trace logs for commands right within your terminal by typing the command followed by an added argument --dev-debug. For example, if your force status command is failing, get the trace logs in your terminal by typing sfdx:force:source:status --dev-debug

Package XML Analysis

To retain the metadata and package.xml file when source push/pull or source deploy/retrieve commands are executed, set an environment variable named SFDX_MDAPI_TEMP_DIR

To set this environment variable on macOS, follow the below instructions.

• To retain the folder only for the current terminal session, type the below command in your terminal . Note here <foldername> is the name of the folder where metadata and the package.xml file will be retained.

export SFDX_MDAPI_TEMP_DIR=<path of your project folder>/<foldername>

• If you want to set this variable permanently for a project, then follow the steps below to set this in your bash profile.

1. In your terminal run nano ~/.bash_profile
2. Add your environment variable using export SFDX_MDAPI_TEMP_DIR=<path of your project folder>/<foldername>
3. CTRL+X to exit the Nano editor and make sure you save your changes. Do not forget to reopen the terminal so your changes apply.

To set this environment variable on a Windows machine, follow the instructions listed here.

SourceMember object records

It’s a good idea to check and make sure SourceMember object records in scratch orgs are created properly. Use the tooling query against scratch orgs to see all SourceMember records and verify that their revision numbers, changedBy, MemberType and MemberName fields are properly populated.

For example, you may run into a problem where a change on the server doesn’t show up in source:status or isn’t pulled down with source:pull. The first thing to do is to query the SourceMember object record for the component you changed. Check to see that it exists and that the RevisionCounter and RevisionNumbers have incremented. If not, there’s a server issue causing the change to not be tracked. If this is the case, it’s likely an issue on the CLI side.

Summary

Hopefully, this post has given you enough information about how Salesforce source tracking works for scratch orgs. One advantage of the source tracking feature of scratch orgs is that it automatically tracks source changes for you so you don’t have to maintain a package.xml file in your source control. Also, you do not have to remember the components that you have changed when building apps.

Check the Salesforce DX Developer Guide for known limitations and workarounds. If you think something is not working as expected, log an issue in the Salesforce CLI GitHub repository.

If you’re interested in knowing more, you can view source code for the Salesforce CLI. The snapshot of the Salesforce CLI source code (in read only mode) is open-sourced here. Also, we are currently enhancing the Salesforce CLI to support development with multiple packages. You can try some of this today and share your feedback here.

Resources

• Trailhead Module – Develop App with Salesforce CLI and Source Control
• Trailhead Project – Quick Start: SalesforceDX
• SalesforceDX Developer Guide
• Salesforce-alm Repository

About the Author

Mohith Shrivastava works as a Senior Developer Evangelist at Salesforce. He is currently focusing on the Salesforce CLI, Platform Services, Communities, and Lightning Web Components. You can follow him on Twitter @msrivastav13