Let’s be honest. The best part of writing tests is seeing them pass. And all too often, the scariest part of running tests is looking through the code coverage numbers. First, you need to find the values you’re searching for and once you do — what do they really mean?
We’re excited to share that over the course of the past few months, we’ve been hard at work revamping your entire Apex experience in the CLI. While the rest of the commands we’ve rolled out were a behind-the-scenes transition for most of you, we are most thrilled about what’s soon to come for the sfdx force:apex:test:run and sfdx force:apex:test:report commands! Our updates will bring you accurate code coverage numbers, cleaned up results, and a new flag to see detailed coverage information. We’ll also give you a preview of more (potentially breaking!) changes to come down the line.
What’s up with my code coverage numbers?
In the past, you might’ve seen inconsistencies in your code coverage data. These issues ranged from seeing differing code coverage numbers across the Dev Console and VSCode, to confusion over the meaning of certain values. With our latest changes, we’ve ensured that all the numbers reported in the CLI after a test run will match the code coverage calculated and seen at the time of deployment. These are the same code coverage numbers you see in Dev Console or when checking the API directly. This means that the values you see in the Apex Code Coverage by Class table below are now accurate. Each row in the table represents the percentage of an Apex class that is covered by all the tests across your entire org. The other noticeable difference is in the number of rows displayed in the table. The table will now only display classes that were directly touched by the test methods in your run.
Note: These coverage percentages come from the Tooling API ApexCodeCoverageAggregate entities.
You may also notice that even the summary table looks significantly smaller than what you’re used to seeing. This leads us to…
Cleaner test run output
The other bulk of our changes focused on trimming down the human-readable output, in order to give you clear, concise information about your test run. We removed values that were redundant or simply too confusing – and after talking to a few of you, we quickly came to the realization that the testRunCoverage value was the biggest culprit. This number was an on-the-fly calculation that did not directly map to any values present during deployment or in the Developer Console. So now it’s out of this view! It will still be available in some of the result files for now.
We also removed the following fields from the overall summary table:
- Passed, Failed, Skipped
- Test Start Time
- Command Time
- Test Total Time
- Hostname
- User ID
And we also added Skip Rate to the view!
A new —detailedcoverage flag
With the rewrite of these commands, we also added a new flag, —detailedcoverage. This flag is designed to be used in conjunction with the —codecoverage flag, and will give you more granular information about the code coverage results for your test runs. Enabling this flag will display a code coverage table that includes a row for each test method that was run, as well as each class that the test method covers. This table will be displayed by default for the sfdx force:apex:test:report command.
Note: These coverage percentages come from the Tooling API ApexCodeCoverage entities.
With this type of granular information, it’ll now be easy to determine whether (and how well) your tests are covering your Apex classes. No more time spent trying to add coverage in various places — you’ll know exactly what and how much each of your tests is covering!
Soon to come: result files and more
Warning: Breaking changes are coming. In the bright-shining future, these same kinds of changes will be made to the output files. Specifically, we’ll be renaming and removing some of the result files that get generated with the output directory functionality across the four different result formats. Until we officially deprecate the current files, we’ll include both versions of the file – giving you time to gradually transition to the new format. These changes do have a potential CI/CD impact, so it’s important to be prepared for them! Here’s a summary of what’s to come:
New filenames
Displayed by default for all result formats:
test-run-id.txt
test-result-<testrunid>.json
test-result-<testrunid>-junit.xml (will be deprecated later this year. Use -r junit flag to get this output file)
sfdx force:apex:test:run -u my@username -n myTestClass -d path/to/dir --codecoverage —detailedcoverage -r <any format> [—detailedcoverage flag]
These default files for code coverage will always be generated, regardless of the specified result format.
test-result-codecoverage.json
test-result-<testrunid>-codecoverage.json (new)
sfdx force:apex:test:run -u myUsername -n myTestClass -d path/to/dir -c -r json [-r json flag]
This will result only in the default files mentioned above.
sfdx force:apex:test:run -u myUsername -n myTestClass -d path/to/dir -c -r human [-r human flag]
This will result only in the default files mentioned above.
test-result.txt (will be deprecated later this year)
sfdx force:apex:test:run -u myUsername -n myTestClass -d path/to/dir -c -r junit [-r junit flag]
These two files will be created in addition to the default files:
test-result-<testrunid>-junit.xml (new)
test-result.xml (will be deprecated later this year)
sfdx force:apex:test:run -u myUsername -n myTestClass -d path/to/dir -c -r tap [-r tap flag]
These two files will be created in addition to the default files:
test-result-<testrunid>-tap.txt (new)
test-result.txt (will be deprecated later this year)
Some changes within the files:
test-result-codecoverage.json
This file displays information for each Apex class covered by the test methods that were run. There’s no change to the format of this file. The data only includes relevant information now. (Similar to the changes described above for the human-readable output) The only classes listed in this file will be ones that are directly touched by the tests that were run. The information comes from the ApexCodeCoverageAggregate entity.
See below for an example entry of an Apex class covered by one of the test methods.
test-result-<testrunid>-codecoverage.json (new)
This file displays information for each test method that was run. This is a detailed breakdown of the results and comes from the ApexCodeCoverage entity. There will be an entry for each test method from the test run.
The following is an entry of an Apex class covered by multiple test methods.
test-result-<testrunid>-junit.xml && test-result-<testrunid>-tap.txt
These filenames are new, but the content is the same! They contain the JUnit and TAP formatted results.
It’s important to realize that this means there will be some breaking changes. You’ll need to edit your CI scripts to intentionally specify -r junit or -r tap by the time that the old style files are deprecated. We’re estimating an August ’21 timeframe for when we’ll have fully deprecated these older files.
Conclusion
Our focus in making these changes is to surface the right data at the right time, so developers can quickly iterate between writing code and writing tests. It’s our belief that the new human-readable output surfaces successes and failures (and relevant code coverage) in an easily consumable way.
Excited about what you see? The refreshed commands are already available in VSCode, and thankfully, you’re just an update away from seeing this in your command line. Run sfdx plugins:install apex to get early access to the features – they’ll be included by default in a future CLI release.
Have feedback or suggestions? Leave us an ‘issue’ here at our GitHub repo.
About the author
Ananya Jha is an engineer for the Salesforce Extensions for VS Code. She started working at Salesforce as an intern and officially joined after graduating from UT Austin in 2018 with a degree in Business Honors and MIS. When she’s not working, she loves dancing and going hiking with her roommates in SF! You can follow her on Twitter.