Introducing the DocumentScanner Mobile Capability

Have you ever taken a picture of a document, business card, or image to save it for future reference? The content of those items, which can sometimes be long, contains data points like a user’s name, email, or other key information that you would want to refer back to. If you want to capture it in Salesforce, it can be a hassle to manually enter it into separate form fields. Wouldn’t it be nice to have that data brought directly into your Lightning web components (LWCs) instead?

We’re excited to announce that in Spring ’24, Salesforce is introducing the DocumentScanner API mobile capability. DocumentScanner enables developers to scan a variety of documents and images, and extract text into structured data that you can use in LWCs, for example, to create or update records. Use DocumentScanner to identify and store key information from documents, such as the name and job title from a business card after a field site visit, text from PDF files, and much more.

Important Note: This plugin will automatically extract data, but in most cases, the extracted data is not 100% accurate and needs user intervention, so it is a semi-automated data extraction process.

Work with the DocumentScanner API

In this section, we’ll discuss how to use the DocumentScanner API. Let’s walk through a quick example of DocumentScanner using Dreamhouse, a sample app that features some LWCs for mobile use cases in a real estate context.

Say you’re a real estate agent who has closed a deal with a client on a new home. The paperwork has been signed, and you would like to create a record of the transaction details on your handy Dreamhouse app. To do so, you need to provide the following Opportunity information:

First Name and Last Name
Email Address
Current Home Address
Phone Number
Sale Price

As you go through the list of items needing updates, you see that it will be painful to enter all of those details manually, especially when you think about your other deals that are closing, too! You need a way to easily extract this information, so that it can be imported much faster into the app.

Enter DocumentScanner! Using the DocumentScanner API, you can identify those specific pieces of object information directly through the LWC.

Configure the DocumentScanner API

The DocumentScanner API enables us to scan the contents of a document. This can be done through two steps:

Configure your LWC HTML template to include a lightning-button Lightning base component (LBC), which will be used to scan a new document.
Connect that button with the JavaScript APIs that DocumentScanner provides to invoke scanning capabilities.

Let’s dive into more details on these steps!

Step 1: Configure your LWC HTML template

For this LWC, we’ll use the .scan(scanOptions) method of the DocumentScanner API, and we’ll need an LWC with a lightning-button base component to invoke it. The DocumentScanner API provides two options for document scanning:

Scan from Device Camera: This allows you to connect to the camera on a device to scan a document in real time
Scan from Photo Library: This allows you to access a previously captured image of a document and make it available for data extraction through DocumentScanner

For our example, let’s add both options. This way, we can scan identification information from the document.

Note: Scan from Device Camera and Scan from Photo Library are two different scanning modes. To invoke both, you’ll need two buttons.

DocumentScanner mobile capability running on LWC in iOS

If there is an error while scanning, or the contents of the document aren’t recognizable, the DocumentScanner API returns an error. The error includes details of the problem, which you can display in your component’s user interface, as seen below.

Once a scan is complete, a snapshot of the scanned document is stored in device memory for the session. You can display a preview of the scanned image directly within the LWC, using the Document data type, or more specifically, the Document.ImageBytes object. Refer to DocumentScanner Data Types for more details.

Step 2: Connect your user interface with the DocumentScanner API

For our Dreamhouse example, we want to capture the document via the device’s camera or from the device’s Photo Library. To do this, we will add the following JavaScript code.

Key DocumentScanner Features

Multi-Language Support

You can configure the language of the DocumentScanner service to support parsing of different languages. We support the following language types as part of the scriptHint object within the options constant: Chinese, Devanagari, Japanese, Korean, and Latin. The Latin language option includes all Latin-based languages, such as English, Spanish, French, etc.

Entity Extraction

Using Google’s MLKit APIs, we have also added the ability to support Entity Extraction. Using the DocumentScanner plugin by itself, you can identify and extract characters in the document (or image), as we have seen above. However, through Entity Extraction, you can support different user actions for the data that is identified within the document scan.

Say, for example, you are scanning a user’s business card, and it contains a user’s email address. Using the entity extraction feature, the Document Scanner will identify that the content’s type is that of an email address value, highlight it to reflect the email address syntax, and enable actions to send emails to the extracted text.

The entity extraction is enabled by using the extractEntities boolean under the options constant, as seen in the JavaScript code above. We support all of the entities that are supported via Google’s Entity Extraction MLKit APIs.

Additionally, similar to multi-language support, entity extraction supports different languages. These include all of the ones supported by Google’s MLKit APIs, including Arabic, Portuguese, English, French, Dutch, German, Italian, Japanese, Korean, Polish, Russian, Chinese, Spanish, Thai, and Turkish. To specify the language type, simply use the entityExtractionLanguageCode in the options constant.

DocumentScanner compatibility

DocumentScanner is available in the Spring ’24 release across the Salesforce Mobile App, Salesforce Mobile App Plus, and Field Services Mobile App offerings. See the Mobile Capabilities Compatibility Summary to stay up to date with its availability across our Salesforce mobile apps.

Conclusion

We hope that you’ve enjoyed this blog post and that you are looking forward to using the document scanning capabilities on your LWC for mobile. To get started:

Dive in! Take a look at our Document Scanner Reference docs and our Example LWCs to get familiar with the plugin.
Personalize it! Take DocumentScanner capabilities and expand on them to customize them for your business needs
Reach out! If you have any questions, comments, or ideas, you can connect with us in the Salesforce Mobile Trailblazer Community

To learn more about Salesforce’s mobile offerings, check out the Mobile and Offline Developer Guide.

About the author

Ashwin Nair is a Product Manager at Salesforce focused on Salesforce Mobile. He is currently working on mobile platform experiences and has been in the web and mobile development space for over seven years. Follow him on LinkedIn.