For a software engineer, there’s nothing more frustrating than a slow, unreliable test suite. My Heroku Connect team had a lot of fast, dependable unit tests, but our end-to-end integration tests with Salesforce were taking up most of our Continuous Integration execution time. In addition to being slow, sometimes they’d fail because of lingering data from previous test runs, or interference from other test runs.
Heroku has a test runner called Heroku CI that’s similar to CircleCI and Travis CI. As part of a recent hack week, I attempted to use Salesforce DX (SFDX) and the new parallel test runs feature of Heroku CI to speed up our CI test suite. We desperately needed to improve our productivity — we were having to wait as long as 25 minutes for tests to complete. The most time-consuming tests were our integration tests for Salesforce-to-Heroku data syncing.
How things used to be
We ran our tests serially. For Salesforce integration tests, we had a pool of nine Salesforce organizations (orgs for short) that were set up with identical metadata. Each test run used a different long-lived org and we figured that with our relatively small team, using nine orgs to cycle through would be enough. But we had to be extra careful about cleaning up after tests since the orgs were being reused.
Scratch orgs
SFDX scratch orgs are disposable Salesforce orgs that are created on demand. They’re typically used for development and testing by Salesforce developers. Even though our tests didn’t involve Apex or Lightning, scratch orgs were still perfect for our use case. We realized that we could create multiple scratch orgs in every test run so that we could execute Salesforce tests in parallel.
SFDX and Heroku CI
When I started to investigate scratch orgs and CI, I found a lot of reference material for Travis CI and Circle CI, but nothing that explained how to integrate scratch orgs with Heroku CI. I’ll take you through all the steps I followed.
To begin, I installed the Salesforce CLI on my local machine and enabled Dev Hub in a Salesforce org. I also had to create a Connected App in the org, configured for JWT-based authorization, so that my CI scripts could create scratch orgs without any human interaction.
I had to refer to two things that got created when setting up the Connected App: the Connected App’s client ID and the server.key file that contained my private key.
The sfdx force:auth:jwt:grant command requires both the client ID and the server.key file to be passed in. How do we provide them during CI execution? Heroku CI allows you to define CI-only config variables in the Settings page of a Pipeline, so I stored the client ID in a config variable called CI_SF_CLIENT_ID. For the private key file, I couldn’t use a config variable, so I needed to encrypt the file as server.key.enc, save it in source control, and then decrypt it while running in CI. Kudos to David Reed for clearly explaining the steps required in a blog post. I then generated an encryption key as described in his blog post and stored it in a CI-only config variable called CI_SECRET_KEY.
My script for creating a scratch org is called create-scratch-org and looks like this:
Once a scratch org is created, we need to obtain an access token and an instance URL so that our tests can authenticate with the scratch org and make Salesforce API calls:
Our tests can now use these environment variables to authenticate with our Salesforce scratch org.
Configuring Heroku CI with app.json
An app.json file is used to define app metadata on Heroku. To use Heroku CI, your app.json file needs to be configured with a test entry in the environments section. Including the salesforce-cli-buildpack will make sfdx and jq available on your app’s dynos. My app.json file includes this section:
When Heroku CI’s parallel test runs feature is enabled, the "quantity": 12 line causes 12 test nodes to run in parallel. Each node has an environment variable called CI_NODE_INDEX that we refer to in our test-setup and test scripts to distribute our test suite across the nodes. And if a test node requires Salesforce, we run our create-scratch-org script to create its own scratch org dynamically! This diagram shows a test run with 12 nodes, six of which are using scratch orgs in their tests.
To distribute our tests efficiently across nodes, we identified the longest running test classes and put them into their own test nodes. We did this manually, but your test runner may be able to support automatic test suite splitting.
Another benefit of having ephemeral orgs is that you don’t have to clean up after yourself. We were able to remove a lot of cleanup code in our Salesforce tests, which helped to further reduce our test durations.
Results
Using SFDX scratch orgs and Heroku CI’s parallel test runs enabled us to cut our total running time down from 25 minutes to under 5 minutes! Successful test runs look like this:
We’re thrilled with the productivity gains we’ve received from SFDX and Heroku CI and hope that you can take advantage of their powerful features too. Your test runs will not only be faster; they’ll also be more reliable. The end result: happier developers!
Trailhead modules
Related posts
- Getting Started with Salesforce DX (Part 4 of 5) by René Winkelmeyer is an introduction to scripting via the Salesforce CLI.
- Getting Started with Salesforce DX (Part 5 of 5) by Philippe Ozil covers Continuous Development and Continuous Integration.
About the author
Arthur Louie is a software engineer at Salesforce, working on Heroku Connect.