In this episode, we sit down with expert Akshata Sawant to discuss MuleSoft Intelligent Document Processing (IDP). Akshata delves into how IDP uses AI to extract and organize data from a variety of unstructured documents, such as purchase orders, invoices, and driving licenses, using AWS Textract for exact information extraction. The discussion highlights IDP’s seamless integration with multiple systems via REST API endpoints and its application in automating the processing of daily invoices received via email. 

Learn about IDP’s ability to handle different document structures without needing multiple templates and the crucial role of human oversight when the AI’s confidence ratio falls below 80%, ensuring accuracy and reliability. This episode offers a comprehensive look at how IDP can revolutionize document workflows, balancing automation with human intervention, and provides valuable resources for further learning.

Show Highlights:

  • Using AI to extract and organize data from unstructured documents
  • Integration of AWS Textract for exact information extraction and seamless integration via REST API endpoints
  • Accessing and using IDP through the Anypoint Platform with ease
  • Managing different document structures without multiple templates and the role of human oversight for accuracy

Links:

Transcript

René Winkelmeyer:

Welcome to the Salesforce Developer Podcast. My name is Rene Winkelmeyer and in this podcast I’m hosting Salesforce Developer Advocates. We’re going to share insightful stories, new tips, techniques and everything actually you have to know these days as a Salesforce developer. Today, I’m very happy to talk on the podcast with Akshata Savant.

Akshata Sawant:

Hello. Hi Rene. Thanks for having me.

René Winkelmeyer:

Speaker 1: Akshara, you have a topic recently discovered, no, I wouldn’t say discovered, but really worked a lot with is MuleSoft Intelligent document processing. What is that?

Akshata Sawant:

MuleSoft Intelligent document processing, we call it IdP and that’s a new tool which went recently in April. And IdP has the power to use… It actually empowers AI. It uses AI and it has the ability to extract the information from the document. Basically any kind of document or any unstructured data can be extracted and streamlined and we can further integrate with different end systems down the line, so it could be any order fulfillment document or any order processing document, purchase order or any text image, even a simple driving license. It has a capability to extract information using AI and organize the data from unstructured documents basically.

René Winkelmeyer:

And which use cases or industries have you seen this before we dive into the technology so that the listeners have an understanding where and actually why you would use IdP?

Akshata Sawant:

Sure. There are a few templates as well available. The one that I’ve most used is the purchase order template. We have this purchase order template when we are placing an order, basically the invoice structure document. That’s what I’ve used. I just had a purchase order, a legitimate purchase order, and I passed this document in IdP. I created a document action and all the information like the invoice detail, the price amount, the transaction details, everything was extracted from the document and under the whole it actually uses AWS Textract to extract the information from the document. But I think currently there’s so much more involvement with the AI part of it as well. There’s so much more on the AI side as well.

René Winkelmeyer:

How do you use the tool? I understand IdP is part of the overall use of solution packet. Is it just like something where I scan something, where I send a PDF, does it have permissions and revenue mechanisms? What’s your overall package of that solution?

Akshata Sawant:

Accessing IdP is super simple. You have to go to any point platform. From there you need to be having proper access rights and IdP is also not currently part of your trial account, but basically you can access it from your Anypoint platform. You can go ahead into IdP. The capability, just like the way we go into Exchange or Design Center or any other rules of capabilities.

And further from there you can just go ahead, you can select what template are you going to upload. It could be any kind of document format, PDF or Word document or image on any kind of image format as well. And that’s how it’s going. You can just scan the document. All the information will be extracted from the document. You can also add in your own prompts if you want particular information to be extracted as per your prompt. Further, you can go ahead and publish this document into any point Exchange. Where it’ll be for the creating a rest API endpoint. And these endpoints are super crucial. You can actually go ahead and integrate this endpoints with any end systems. I hope that gave you a clear idea of what IdP does and how can you access IdP.

René Winkelmeyer:

I think that that is something that I want to dive deeper into what you just mentioned with auto-publishing, the document type definition, if that’s the right word, how to reuse it actually with an Exchange or wireless APIs but I’d like to get maybe one step before, when you look at IdP uploading, this makes a lot of sense if you have mass of documents, it enables us to have three documents a quarter, it doesn’t really make sense I guess.

Akshata Sawant:

Absolutely. I can give you an example here. Sorry to cut you in because I’m really excited about this tool. An amazing example would be suppose if you’re getting daily invoices from customer over your email, so what you can do is you can set some filter or download these attachments from your email and with the help of MuleSoft of course using the MuleSoft connectors and then further you’re going to send these documents to IdP with the help of REST API. IdP, we also have the connectors in MuleSoft using which you can send documents to IdP and process the documents. Whenever you’re scanning the document which has been downloaded from the Gmail, it’ll scan all the information, extract all the information from your invoice, it’ll process that. Further, you can also process this information with the help of Dataview transformation and stuff like that. And you’re good to go ahead and build your full-fledged integrations with IdP.

René Winkelmeyer:

It basically means, for example, whatever I have my service at winkelmeyer.com account, simple example, maybe not the best, and that’s where a ton of different PDF documents got sent to or Word documents, I would have an automation running that attack fetches those documents, extracts the data. It’s basically like OCR, right?

Akshata Sawant:

Yeah. It’s basically. I think there is some certain difference between the OCR and the IdP because IdP is more of AI powered. We are also bringing in the trust layer in the picture because we are passing data, we extracting data, so we need to ensure that it’s secure as well. And other than that, it’s a bit different from OCR, so I think we should go ahead and give it a try. It’s super interesting because that’s currently my favorite tool that I’ve been trying and exploring all about.

René Winkelmeyer:

But that means just as I understand this, whatever I get the invoice from Ashada.com, it’s extracted and then it’s an IdP and then automatically trigger something, or is that, for example, if I want to invoke something in my ERP system based on the just detected invoice, how would that work?

Akshata Sawant:

You can do that. For that, you can build a Mule Flow in which the first step would be downloading the attachment or the invoice from your email akshata.com. And then you’re going to extract all this fields with the help of IdP connector, which could be post document action. You’ll get the post document action would be sending the document to IdP. That’s a post endpoint that the IdP has released on Exchange. And then you can get the document that has been extracted. You’ll be getting a document ID for the one that we have uploaded just now into IdP. With the help of document ID, you can get all the information that was extracted from the document. It could be my name, email address, my shipping address, my invoice details, transaction details and everything. And once we have all this information, we can further go ahead and use any other connector in our Mule Flow that could be triggering another system as well based on the information, so it could be series of logical sequences, connectors, and we can just build a flow or maybe our entire use case as per our requirements.

René Winkelmeyer:

That sounds pretty cool. Is integration quote-unquote limited to just mules of Exchange or for example, if I want to use it with RPA or with Salesforce Flow, how would that work?

Akshata Sawant:

It would work exactly the similar way. You can also integrate the document action that you have published in the Exchange. You can further go ahead and integrate it with RPA, you can integrate it with Flow and there’s so much more as well coming. We are also having plans of bringing in the Einstein factor into IdP itself. That is currently in beta and we are trying to… It might be in GA, I hope we are having the clause here, the forward-looking statement. It’s expected to be December around, so we are going to bring in more Einstein part into it as well.

René Winkelmeyer:

That means, for example, if I want to use that data from, let’s say, the invoice within the Salesforce Flow and do something like that in Mark, is there a direct integration or is that an external server that I would use?

Akshata Sawant:

There is a direct integration, just like the way we connect with different connectors in the Flow or recently I’ve done this integration, I haven’t actually tried the direct integration with IdP and Flow, but I have done with the MuleSoft APIs and Flow. The thing that will be publishing in exchange the document action, similar document action, we can access in the Flow as part of action or method and we can call it directly, so there’s no extra configuration that is required. It’s just simple Flow integrations and we can invoke IdP.

René Winkelmeyer:

And how would they work with RPA, robotic process automation?

Akshata Sawant:

Process automation, yeah. Similar to that, we can also create an entire process automation flow in RPA and you could be having some tasks which you want to be automated, the daily repeated tasks, so you can get them started with RPA, setting up the process and everything.

René Winkelmeyer:

Yes. I have one more question. Maybe two, or three or four.

Akshata Sawant:

Sure.

René Winkelmeyer:

One question that comes to my mind is, and you just mentioned that templates. Let’s say, for example, a purchase order template, people are sending various different versions of documents as we know. Would I have to configure all of those or is IdP smart enough to detect what it is so they don’t need like 25 different templates for purchase orders?

Akshata Sawant:

Yeah. IdP is actually smart enough to identify the document structure and extract information and place it in a proper placeholder. It has more than required fields actually in a particular template, so at the moment there are three templates, which is purchase order, invoice and generic template. In purchase order as well, there are several fields like buyer details, seller details, transaction details, and around 50 to 60 fields.

But a document may or may not be having all these fields, there might be few fields which are omitted. You can actually skip them out of the document or if you want to add another field which you think is not a part of IdP, you can add that field as well. And Einstein is going to play the magic here and it’s going to extract the field for you based on what placeholder you’re adding to a new field. But usually that’s not required. The template that IdP provide, it actually has a lot of sufficient fields basically to map your payload, map your fields correctly, and in future we expect it to bring in more such templates, more such relevant templates to scan your documents accurately and map them correctly as well.

René Winkelmeyer:

Another question that comes to my mind, that’s my old business process consultant mindset I guess is just when, again, Winkelemeyer.inc gets 500 different documents per day and I want to use all the full information and basically don’t want to have to worry about any of that, but then there may be something that is coming that won’t be detected because some German company, because I’m German, is going to send a German invoice or whatever, so how would that work in this fully automated Flow, which is fully REST enabled as you just described? How would I have to work with that?

Akshata Sawant:

I was actually about to mention this thing and thanks for bringing this up. When talking about the AI part, the human in loop also comes into the picture and that’s where we can say that… What do you call it? When you’re working with IdP, so suppose if you’re having a German that you mentioned, the German document or a German stuff that’s coming in and IdP is not sure about it, so there’s going to be a confidence ratio.

If IdP detects a field or detects or input and output stuff and the confidence ratio is below 80%, it’s going to ask a human to review the document. Whenever you’re a document or creating a document action, you’re actually going to set up a reviewer as well, so whenever there is less confidence ratio, IdP is not sure if you still have to go ahead with the field, it’s going to ask a human to review it. And only then the document status or document process status is going to change from progress to processed. Otherwise, you cannot fully process the document. I hope that was the one that you asked about.

René Winkelmeyer:

Absolutely. And I think that’s something that you just called out really well as a human in the loop.

Akshata Sawant:

That’s always going to be in the picture and that’s what we have been emphasizing at Salesforce as well, right?

René Winkelmeyer:

Yes, exactly. I think that’s really important. There’s a certain degree of automation, but with all technology where AI supports us as humans is also to make sure we are there, called in to make the decision at the end of the day. Cool. Akshata, I think you also published a blog and a video, which I’ll put in the show notes so that people also can read and listen and view actually what IdP does.

Akshata Sawant:

I have a couple of blogs and I’ve recently published two videos and there are so many more coming. Also, we have done a few live streams and there are so many resources and I think we’ll be listing them down here.

René Winkelmeyer:

Absolutely. Akshata, thank you for being my guest today. I learned a lot about IdP. Maybe if I have some time, I’m also going to try it.

Akshata Sawant:

You should. I’m sure you’re going to love it.

René Winkelmeyer:

Cool. Thank you very much. Have a good day.

Akshata Sawant:

Thank you so much. Bye.

René Winkelmeyer:

If you’re still here, thanks for listening and head over to developer.salesforce.com/podcast to check out all our episodes and also check out the show notes because that’s where you will find all the great links to provide you more information and insights about all the topics we talked about today. Bye-bye.

 Get notified of new episodes with the new Salesforce Developers Slack app.