This is a supplementary post, related to our Getting Started with Packaging and Modular Development series.

In the first post of that series, we walked through an example of retrieving metadata in an unmanaged package and converting it into a format compatible with Salesforce DX. But what about metadata that isn’t available for extraction using that method? What does that workflow look like? What resources are available to help you figure out what metadata is even available for extraction?

That’s what we’re going to talk about in this post.

How can you see what’s missing?

Not every piece of metadata is currently available in the screens you can access via Setup. The Package Manager (AKA the place where you compose unmanaged packages) and change set UIs fall into this category. So how do you discover what metadata is really out there, and what metadata can still be retrieved?

My first stop is the SOAP API Developer Guide. It is the most comprehensive list of sObjects, their purposes and fields, and—most important for our purposes—the operations you can perform on them. Another great feature is the Data Model section, which has ERD diagrams (visualizations of objects and relationships) for a large number of built-in platform services. These diagrams can help you identify what sObjects are involved with the functionality or customization you’re trying to extract. For example, this is the ERD for working with Knowledge:

Just some of what’s included in the SOAP data models: Sales Objects, Content Objects, and of course, Salesforce Knowledge Objects.

As I worked on creating a Salesforce DX project with metadata I’d grabbed from an org via unmanaged package, I realized the metadata related to Einstein Bots my team and I had built hadn’t been pulled into the package. I double-checked the Package Manager menu, and didn’t see any obvious options. So I needed to determine if I could grab that metadata another way, like interacting directly with the Metadata API, or if that metadata would need to be manually re-created in other environments.

So I headed over to the SOAP API docs to see what sObjects I might need to work with and see what my options were.

Which objects do I really need?

If you’re lucky, when you go to the SOAP API docs, you’ll see the objects you’re working with included in one of the data model ERD diagrams. You can use that as a guide to then dive deeper into the steps I’ll talk about below. But if you, like me, don’t see the functionality you’re working with in any of the data models, you’ll need to investigate objects on your own.

I knew from the process of building bots and deploying them that part of the metadata involved Live Agent functionality. So I started with that section of the docs. I looked at the descriptions of the objects first. I focused on whether an object represented configuration details about my org (like the LiveAgentDeployment object) or whether it represented details about what my bot did when customers tried interacting with it (like the LiveAgentTranscript and related objects). I only wanted configuration objects. If I wasn’t sure about an object, I added it to a list I was building of objects I might be interested in.

The next thing I looked at was the actions I could perform like: query() , create(), update(). The action that I needed to see listed was retrieve() If you can’t retrieve() a piece metadata…then you can’t retrieve a piece metadata.

Once I had a list of objects that I thought I might need to grab, I needed to confirm whether or not I was on the right track. A quick sanity check is to see what objects aren’t currently supported by the Metadata API.

Once I’d checked my objects against the list of supported metadata, I logged into Workbench and ran queries against the objects I was interested in (for the objects that were queryable—which all of mine were). I looked at the results of the queries to better understand what pieces of my org configuration each object actually represented. Here’s an example query against the LiveChatDeployment object:

From there, I added objects that looked relevant into my package.xml — the list of metadata I wanted to grab through the Metadata API.

Building your package.xml

Depending on the IDE you use, you might be able to simply take the list of objects you’re interested in and fetch them without building a package.xml yourself. There is built-in functionality to do this in Eclipse, if you’re using the Force.com IDE plug-in.

If you’re not using an IDE that will do the work for you, you’ll need to create a package.xml yourself. If you’ve worked with the ANT Migration Tool or Metadata API deployments, you may already know how to compose these files. If you don’t already have example files you can work with, you can find examples of the syntax in the Metadata API documentation.

I threw together my small file quickly:

Once you have a package.xml, you can do a retrieve from the command line using the Salesforce CLI. Instead of using the -p parameter (like I did for my unmanaged package retrieval), you can use the -k parameter to have the CLI fetch the metadata in your package.xml file.

While this was the technique I ultimately used to get extra metadata into my project, I don’t use this approach while still experimenting with my package.xml.

I prefer to use Workbench for this phase. Workbench is a more useful tool at this stage of the process because the ability it gives you to examine all the messages returned by the Metadata API, rather than success/failure messages, which is what you’ll see in the output from the CLI. At this stage, you need more granularity than whether or not your call was successful.

When I ran my initial retrieval, using the markup above, it was successful. But when I looked at the actual .zip file, none of the metadata I requested actually showed up. Digging into the responses from the Metadata API, I saw a common error message: Can’t retrieve non-customizable CustomObject.

So even though these objects were supposed to be retrievable, they actually weren’t. I could also see from the error that I didn’t need to worry about losing customizations because of this behavior. Because these objects weren’t “customizable” (i.e. you can’t add fields or change field attributes), our org’s usage and customization of these objects just lived at the level of data, not metadata. So I didn’t need to worry about losing metadata if I left those objects out of my package.

As I worked on this extraction puzzle, the orgs I was working with started to change from API 42.0 to API 43.0. When one of my colleagues, René Winkelmeyer, reminded me to check the new Metadata API docs, I saw that I had more options to investigate for retrieving our bot configurations — specifically, the Bot and BotVersion sObjects.

After following the same steps above, I drafted a final package.xml (with some Apex classes related to my bot added in). It ended up looking pretty simple:

Note: the Bot and BotVersion markup are interrelated. For each bot, you’ll receive a file with .bot as the extension, with the <Bot> and <BotVersion> markup within the same file. You have to request both objects in your package.xml to retrieve the metadata correctly.

Now, I just had to wait for the org I needed to extract from to be upgraded to the latest API. Once it was, I could use the Salesforce CLI to retrieve my bot using the .xml file above. I could then convert it into a Salesforce DX compatible format and add it into my project.

Summary

Not all metadata is available for retrieval through the Package Manager interface in Setup. In these cases, you’ll need to work directly with the Metadata API and create a package.xml to retrieve the metadata you’re after.

If you’re looking for a more about package.xml files and options for using those files in your deployments, head over to Trailhead and get hands-on with the Application Lifecycle Management module.