Working with Modular Development and Unlocked Packages: Part 3

This is the third installment in a series exploring how to begin incorporating packages into your app development lifecycle, and what adoption may mean for your team’s change management and release processes. Over the course of this series, we’ll talk about:

• Part 1: What even is a package, anyway? How can you start to experiment with segmenting your org?
• Part 2: How can you start to organize metadata from an app, let alone an entire org, into packages? How do you tackle organizing your metadata and projects in source control?
• Part 3: What do these changes mean for app builder workflows? What will happen if I install an unlocked package into my production org today?
• Part 4: How can you define a successful Git branching strategy that works best for most team sizes? How, when and where should packaging be added to your continuous deployment?

In this post, we’ll look at building unlocked packages and working with versioning, as well as setting up dependencies between packages.

In the last post, we discussed how we approached segmenting an app into package modules. If you haven’t read that post yet, it’s a good idea to go check it out and come back here when you’re caught up.

Turning modules into packages

As I built, I could control which modules deployed and the order they deployed by modifying my sfdx-project.json. If I didn’t want a module to deploy, I removed its entry from the .json. When I wanted to deploy modules in a certain order — simulating my package installation order and testing my dependency management — I added individual modules into my sfdx-project.json one-by-one, and ran sfdx force:source:push in between each addition.

Once I felt reasonably confident in my deployment order and modules, I needed to turn my modules into packages.

Before you start to experiment with package creation and versioning, be aware that in Summer ’18, you can update the metadata in an unlocked package, but you cannot delete an unlocked package. Because I was using a dev hub intended for experimenting with packages, I had the luxury of not worrying that I was going to clutter up my team’s working environment with discarded packages. But even with that luxury, just three of us working on a handful of packaging experiments meant that a “simple” command like sfdx force:package:version:list returned quite a bit of junk. If you’d like to experiment with packaging, but don’t want to put those experiments into your production Dev Hub, I recommend using a trial Dev Hub environment. (Or, if you’re reading this after Summer ’18, go check and see if you can delete or deprecate unlocked packages.)

When it’s time to generate your packages, you’ll issue a series of force:package:create commands. During this step, you’ll want to pay attention to the location of the modules you want to package within your project’s root directory. At a high level, my project looked like this:

|____Easy-Spaces
| |____.sfdx
| |____config
| |____data
| |____es-base-code
| |____es-base-objects
| |____es-base-styles
| |____es-images
| |____es-space-mgmt

Only four of these modules would become packages: es-base-objects, es-base-code, es-base-styles, and es-space-mgmt. To create my first package, for my es-base-objects module, I issued the following command:

sfdx force:package:create -n ESBaseObjects -d 'Easy Spaces base objects' -t Unlocked -r ./es-base-objects

Highlights:

• Package type: When you create a package, you are required to say which package type you want to create. The -t Unlocked part of my command creates an unlocked package.
• Path: The -r parameter allows you to specify a relative path to your package contents. This allows you to issue multiple package create commands without having to change directories. It also helps your final sfdx-project.json be more readable.
• Description: The -d parameter is how you provide a description for your package. Use it!

You’ll issue an sfdx force:package:create command for each of your package modules.

Using packages in your DX project

After each of my force:package:create commands, the CLI updated my sfdx-project.json (for me! really! automagically delicious!). After the updates, it looked like this:

{
    "packageDirectories": [
        {
            "path": "es-base-objects",
            "default": false
        },
        {
            "path": "es-base-code",
            "default": false
        },
        {
            "path": "es-base-styles",
            "default": false
        },
        {
            "path": "es-space-mgmt",
            "default": true
        },
        {
            "path": "./es-base-objects",
            "package": "ESBaseObjects",
            "id": "0HoB00000004CWSKA2",
            "versionName": "ver 0.1",
            "versionNumber": "0.1.0.NEXT",
            "default": false
        },
        {
            "path": "./es-base-code",
            "package": "ESBaseCode",
            "id": "0HoB00000004CWXKA2",
            "versionName": "ver 0.1",
            "versionNumber": "0.1.0.NEXT",
            "default": false
        },
        {
            "path": "./es-base-styles",
            "package": "ESBaseStyles",
            "id": "0HoB00000004CWcKAM",
            "versionName": "ver 0.1",
            "versionNumber": "0.1.0.NEXT",
            "default": false
        },
        {
            "path": "./es-space-mgmt",
            "package": "ESSpaceMgmt",
            "id": "0HoB00000004CWhKAM",
            "versionName": "ver 0.1",
            "versionNumber": "0.1.0.NEXT",
            "default": false
        }
    ],
    "namespace": "",
    "sfdcLoginUrl": "https://login.salesforce.com",
    "sourceApiVersion": "43.0",
    "packageAliases": {
        "ESBaseObjects": "0HoB00000004CWSKA2",
        "ESBaseCode": "0HoB00000004CWXKA2",
        "ESBaseStyles": "0HoB00000004CWcKAM",
        "ESSpaceMgmt": "0HoB00000004CWhKAM"
    }
}

Highlights:

• packageAliases: This attribute, created for you by the CLI starting in Summer ’18, allows you to use a simple name to refer to a specific package in various force:package commands. This makes commands like versioning much, much easier. This capability also helps for those looking to incorporate automation into their package development processes. The name you choose for the package (the -n parameter in your create command) will be used as the package alias.

• versionName & versionNumber attributes: Creating a package is just the first step. In order to get something inside your package, you have to create a version of the package. Versions are snapshots of your package contents at a point in time. The Salesforce CLI needs a naming convention to follow when creating the versions of a package, including a version name and a format to use for version numbering. The values in the example above were provided by the CLI, as a part of the force:package:create response. If you choose to override these values, be mindful about the formatting of the ‘versionNumber’ attribute. You’ll need to follow the Major.Minor.Patch.Build convention. You can use a relative value like NEXT only in the final (build) position.
• Lines 3-18: These lines are the original entries for my package modules. The CLI updates to your sfdx-package.json are only additive. Keeping these lines (which are now redundant) in your sfdx-project.json as you work on versioning won’t necessarily cause any problems — but keep in mind you can only have one package directory with the ‘default’ attribute set to ‘true’. You’ll also want to remember to clean up your sfdx-project.json at some point.

Now that we’ve created packages and have the updates in our DX project, we should be all set to create some versions of our packages, right? Not quite yet.

Let’s talk dependencies (between packages)

When you successfully issue a force:package:create command, that doesn’t mean that your package structure is solid. The real test for the contents of your package is when you attempt to create a package version.

For my ESBaseObjects package, which has no dependencies, versioning was simple:

sfdx force:package:version:create -p "ESBaseObjects" -x

However, continuing to try and create a version of my ESBaseCode package resulted in a whole bunch of errors:

=== Package Version Create Request
NAME                           VALUE
─────────────────────────────  ──────────────────
ID                             08cB00000004CsYIAU
Status                         Error
Package Id                     0HoB00000004CWXKA2
Package Version Id
Subscriber Package Version Id
Tag
Branch
Created Date                   2018-06-07 09:35
Installation URL
=== Errors
(1) testDataFactory: Variable does not exist: m
(2) testDataFactory: Variable does not exist: markets
(3) testDataFactory: Invalid type: Space__c
(4) testDataFactory: Invalid type: Space__c
(5) testDataFactory: Variable does not exist: s
(6) testDataFactory: Variable does not exist: s
(7) testDataFactory: Variable does not exist: s
(8) testDataFactory: Variable does not exist: s
(9) testDataFactory: Variable does not exist: s
(10) testDataFactory: Variable does not exist: s
(11) testDataFactory: Variable does not exist: s
(12) marketServicesTest: Variable does not exist: allSpaces
...

The more detailed list of errors returned by the query revealed a set of common key issues:

marketServicesTest: DML requires SObject or SObject list type: List<Space__c>
marketServicesTest: Invalid type: Market__c
marketServicesTest: Invalid type: Space__c
testDataFactory: Invalid type: Space__c
testDataFactory: Invalid type: Reservation__c
testDataFactory: Variable does not exist: Reservation_Status__c
marketServices: Invalid type: Space__c
marketServices: Invalid type: Market__c
marketServices: Invalid type: Schema.Space__c
marketServices: Invalid type: Schema.Market__c
marketServicesTest: Invalid type: Market__c
marketServicesTest: Invalid type: Space__c
marketServicesTest: DML requires SObject or SObject list type: List<Market__c>Customer_Fields.Contact_Customer_Fields: In field: Customer_Status__c - no CustomField named Contact.Reservation_Status__c found

Every piece of my ESBaseCode package that explicitly referenced the Easy Spaces schema became an issue at the time of package versioning. This is because my ESBaseCode package is dependent on my ESBaseObjects package — but the CLI didn’t know that.

In order to generate a version for my ESBaseCode, I needed to update to the part of my sfdx-project.json describing my ESBaseCode package to add in its dependency on the ESBaseObjects package:

{
     "path": "./es-base-code",
     "package": "ESBaseCode",
     "id": "0HoB00000004CWXKA2",
     "versionName": "ver 0.1",
     "versionNumber": "0.1.0.NEXT",
     "default": false,
     "dependencies": [
         {
           "package": "ESBaseObjects",
           "versionNumber": "0.1.0.LATEST"
         }
     ]
},

Similar to how the ‘NEXT’ keyword operates in the ‘versionNumber’ attribute, you can use the ‘LATEST’ keyword to link your dependency to the newest build of a package version. After including the dependency, I could create a version of my package.

My full sfdx-project.json, with all the dependencies added (and with the redundant entries deleted), looked like this:

{
    "packageDirectories": [

        {
            "path": "./es-base-objects",
            "package": "ESBaseObjects",
            "id": "0HoB00000004CWSKA2",
            "versionName": "ver 0.1",
            "versionNumber": "0.1.0.NEXT",
            "default": false
        },
        {
            "path": "./es-base-code",
            "package": "ESBaseCode",
            "id": "0HoB00000004CWXKA2",
            "versionName": "ver 0.1",
            "versionNumber": "0.1.0.NEXT",
            "default": false,
            "dependencies": [
                {
                "package": "ESBaseObjects",
                "versionNumber": "0.1.0.LATEST"
                }
            ]
        },
        {
            "path": "./es-base-styles",
            "package": "ESBaseStyles",
            "id": "0HoB00000004CWcKAM",
            "versionName": "ver 0.1",
            "versionNumber": "0.1.0.NEXT",
            "default": false,
            "dependencies": [
                {
                "package": "ESBaseObjects",
                "versionNumber": "0.1.0.LATEST"
                }, 
                {
                "package": "ESBaseCode",
                "versionNumber": "0.1.0.LATEST"
                }
            ]
        },
        {
            "path": "./es-space-mgmt",
            "package": "ESSpaceMgmt",
            "id": "0HoB00000004CWhKAM",
            "versionName": "ver 0.1",
            "versionNumber": "0.1.0.NEXT",
            "default": true,
            "dependencies": [
                {
                "package": "ESBaseObjects",
                "versionNumber": "0.1.0.LATEST"
                }, 
                {
                "package": "ESBaseCode",
                "versionNumber": "0.1.0.LATEST"
                },
                {
                "package": "ESBaseStyles",
                "versionNumber": "0.1.0.LATEST"
                }
            ]
        }
    ],
    "namespace": "",
    "sfdcLoginUrl": "https://login.salesforce.com",
    "sourceApiVersion": "43.0",
    "packageAliases": {
        "ESBaseObjects": "0HoB00000004CWSKA2",
        "ESBaseCode": "0HoB00000004CWXKA2",
        "ESBaseStyles": "0HoB00000004CWcKAM",
        "ESSpaceMgmt": "0HoB00000004CWhKAM"
    }
}

The order that you list the dependencies (lines 71-74, for example) should follow your intended installation order.

As you work with creating package versions, keep a few things in mind:

• Flow/FlowDefinitions: The Salesforce CLI will attempt to ignore any inactive flow metadata in your project by default. However, if you do not have any active versions of a flow, this behavior will cause your attempt to create a package version to fail, as well as give you a not-very-useful error message. I highly recommend you double-check that any flows are active and proactively remove any inactive flow version metadata from your package directory before attempting to generate a package version.
• JSON output: I also recommend using the -json parameter if you run into any issues with a force:package:version:create command. For example, the above issue with flow resulted in an error that simply read: ‘ERROR: Cannot read property ‘0’ of undefined.’ Using the -json parameter will give you the path to the file in your CLI that actually threw the error, so you can trace the logic back to the a more specific area of your project. (Empty files in your project directories in general may also cause the CLI to give you this error.)
• Querying packaging-related sObjects from the CLI: If a package version command fails, the CLI will return a message suggesting you run a query against the ‘Package2CreationRequestError’ object. The CLI will also provide the text of the command for you. However, if you simply copy/paste the command and run it, you will get an error message about querying for an invalid type. This is because the default behavior of the CLI is to run all force:data:query commands against your default scratch org. To run the query against your project’s Dev Hub, you need to add the -u parameter to the command. You’ll want to add the -u parameter for any ‘Package2’ related object queries.
• Package deprecation: You cannot, as of Summer ’18, deprecate an unlocked package version. Be aware of this when starting to experiment with packaging. Make use of the -d parameter to give your packages meaningful descriptions when issuing a sfdx force:package:create.
• Package version name: Just as adding descriptions to your package is a best practice, replacing the default value created by the CLI with a meaningful package version name will also help you better manage your packages in the long run. You can also add descriptions to your package versions.
• Package version limits: Be mindful of the number of package version requests you’re allowed in a 24-hour period. Use sfdx force:limits:api:display (with the -u param pointing towards your Dev Hub) to track your usage.
• Package version aliases: The only way to have the CLI automatically update the ‘packageAliases’ section of your sfdx-project.json with an alias for a package version is to use the -w parameter (and provide a reasonably large wait time window) when you create a package version. If you run a force:package:version:create command without this parameter, or if the wait interval times out before your package version gets created and you’d like to be able to use an alias for commands like force:package:install, you can manually add an entry into the ‘packageAliases’ section of your sfdx-project.json. The alias should follow a “packageName@versionNumber” format, and map to an ID beginning with ’04t’. (It’s the value you’ll see listed under ‘subscriber package version ID’ when you run sfdx force:package:version:list.)

After you’ve generated versions of your packages, you can validate them by installing the packages in a clean scratch org or sandbox. You won’t be able to install your package into production until you run sfdx force:package:version:promote. There are some things to consider before taking this step, which I address below.

Impacts on app dev

We touched on the differences in application development with unlocked packages versus traditional deployment tools (metadata deployments, change sets, etc.) in the first installment of this series. But what does the app dev lifecycle look like for orgs that use packages? What happens if you decide to install an unlocked package into production today?

Unlocked packages are currently in beta, which means there are limitations around what you can do with unlocked packages right now. It’s important to understand what the current functionality of unlocked packages is and be aware of any gaps that might impact your ability to manage your org effectively. For example, you cannot use the Lightning App Builder to customize or edit Lightning Apps that have been installed as part of an unlocked package right now. As mentioned a couple times, deprecating package versions is on the roadmap for unlocked packages, but isn’t part of Summer ’18 functionality.

So what does all this mean for app development right now? Should we all just wait until unlocked packages are GA?

Again, the right answer depends on what’s right for your org. As we talked about in the first part of this series, being successful with packaging means also developing strong systems for source control and beginning to experiment with modularizing your org’s metadata. So for teams that want to get started on the road to packaging, they may choose to get started now. They may start by trying to build some small modules that could become packages in order to work with source control and metadata separation techniques.

And when they feel confident in their modularization approach and source structures, they may even decide to incorporate package versioning and package installation into their app development lifecycle. They may also decide to only promote (i.e. install) their packages as far as their final staging sandboxes.

Incorporating packaging into the app dev lifecycle in an intentional and staged manner means more of the team supporting app development can get a sense of how packaging will affect their work. Team members can provide feedback about changes in package structure to the group working on the first packages for the org and unwanted side effects can be identified early, without affecting users in production. The team can also start to figure out an effective system for capturing and syncing changes made in production with source control systems.

As the functionality of packaging changes, the team can make incremental adjustments to their packages along with these changes. (And the need to understand how a something works and what it can and can’t do effectively doesn’t disappear when it goes GA, as we all know…right?) And as the team makes these incremental changes to their packages, they’re also building up experience and systems to support their packages. When the team decides a package is ready to install into production, they can do so based on real knowledge of their systems and their packages.

What’s next and what to do now

In our next installment, we’ll walk through how to incorporate packaging into your source control workflows and how to start thinking about impacts of packaging on continuous integration and delivery systems.

In the meantime, now is a great time to get hands on with Salesforce DX. You can sign up for a free 30-day trial org here that will give you a dev hub to experiment with. You can also explore the packages used in the examples above and throughout this series by checking out the Easy Spaces application.

You can also dive deeper into Salesforce DX and unlocked packages on Trailhead:

• Quick Start: Unlocked Packages
• Understand Salesforce DX Development Model
• App Development with Salesforce DX