Einstein Vision helps you build smarter applications by using deep learning to automatically recognize images. It provides an API that lets you use image reco
gnition to build AI-enabled apps. Today we’ll explore how to access Einstein Vision from Scala, which is a scalable language program.
Scala is a language that can be used on shell with one-line commands, or it can be used to build complex mission-critical languages. It’s becoming popular with programmers who want a robust language and libraries but don’t want to give up the convenience of a Java virtual machine (JVM).
Some features of Scala Language include:
- Functional programming language
- Object-oriented features
- JVM-compliant
Because visual data is expanding rapidly at a 9% CAGR (compound annual growth rate), it’s vital for organizations to tap into nontextual data to discover conversations about their brand.
Some use-cases to consider include:
- Tracking your brand perception on social channels like Twitter and Facebook
- Giving field service agents the ability to identify and classify objects, such as factory equipment, to handle service updates faster
We cover details about algorithms used in Einstein Vision service in a separate blog post, Using Einstein Within Golang.
How does Einstein Vision work?
Einstein Vision is part of a supervised learning technique in which a model is trained on pre-labeled training data. The training dataset consists of labeled images and is loaded into Einstein via an API call. Next, a REST API call is made to train the dataset, and the output is a trained model, which has a unique model ID. An API call is made to predict the label of a new image that the model has never seen before. You can also find out metrics on accuracy of the model. This diagram shows the schematics of working with Einstein Vision service.
Scala Einstein library
In this section, you learn about the class structure and various functions exposed by the Scala library that can help you access the Vision API REST endpoints.
The class structure
The library has been divided into multiple packages with the base package being com.einstein.lib.
The main vision service methods are in com.einstein.lib.VisionService and com.einstein,lib.DataSetService.
Complete class hierarchy:
Vision service
This service trait is used to predict the label of an image residing in a local path or remote url:
Dataset service
This service trait (interface in Scala) provides life cycle methods of the dataset and model:
Code examples
This section explores how dataset is created, used for training, and deleted.
Create a new dataset
Create a new dataset by providing the url of the zipped dataset folder with labels and images.
The following method createDataSet is defined in DataSetServiceImpl class:
Let’s look at an example code that uses this method. As you can see we are creating a new dataset from a zip file located at a URL by calling the the method createDataSet(..)
Train a dataset
Create a new dataset by providing the url of the zipped dataset folder with labels and images.
The following method trainDataSet is defined in DataSetServiceImpl class. This method takes accessToken and datasetId and the parameters.
Create a new dataset by providing the url of the zipped dataset folder with labels and images.
The following method trainDataSet is defined in DataSetServiceImpl class:
1. Instantiate accessToken from environment variable.
2. Create datasetId variable with the appropriate value.
3. Instantiate DataSetServiceImpl.
4. Call dataSetService.trainDataSet(.., ..) with appropriate parameters.
Predict label of an image
Let’s look at how to predict an image using this library. We’re going to use two methods defined in VisionServiceImpl. Method predictImage() is used to predict remote image located at url. If the image is located in a local path, the library provides another method predictLocalImage.
Implementation of both these methods can be found as below.
Predict remote image:
Code walkthrough for predictImage() method:
1. Create org.apache.http.client.HttpClientinstance, we are using Apache’s HTTP library.
2. Create org.apache.http.client.methods.HttpPost Object using Vision Prediction url
3. Add HTTP headers Cache-Control and OAuth access token using Authorization key.
4. Add HTTP body using org.apache.http.entity.mime.MultipartEntityBuilder.
5. This body contains key, values for sampleLocation and modelId.
6. Create a HttpResponseHandler using BasicResponseHandler instance.
7. Execute the client.
8. Extract the body from HttpResponse object and parse JSON in the body using GSON library.
9. Extract the List of probabilities and return the method call with the List object.
Predict local image:
Using the predict image APIs
Local image prediction example
In the next example, a remote image specified by localfilePath is classified using a preexisting model,modelId=FoodImageClassifier
Code walkthrough:
1. In the main method of the object PerdictLocalImage define variable localFilePath for the local path of image file for which label needs to be predicted.
2. Instantiate variable modelId model to be used, FoodImageClassifier in this case.
3. Instantiate accessToken variable, defined in the environment key access_token
4. Instantiate VisionServiceImpl object visionService.
5. Call predictImage(...) method on the visionService object.
6. Iterate over returned list of probabilities and print label and corresponding probability.
Label for the food image, samosa, in this case, being predicted is shown below:
Output of the program listing is:
Remote image prediction example
In the next example, a remote image specified by a url is classified using a preexisting model, modelId=FoodImageClassifier.
Code walkthrough:
1. In the main method of the object define PerdictRemoteImage variable for url the image to be predicted.
2. Instantiate variable modelId model to be used, FoodImageClassifier in this case.
3. Instantiate variable accessToken, Instantiate object defined in the environment variable access_token.
4. VisionServiceImpl visionService.
5. Call predictImage(...) method on the visionService object.
6. Iterate over returned list of probabilities and print label and corresponding probability.
Food image being predicted, pizza, in this case, is shown below:
The output of the listing for remote image prediction:
Compiling and running the samples
In the library sbt tool is being used for compiling the samples. Please review the build.sbt file below which lists the library dependencies
You can compile and run the samples using the following commands, please note PredictLocalImage is just an example
Summary
This blog post provides you with the basics on how to use Einstein Vision from a Scala stand-alone application. Hopefully, these samples provide a good starting point for developers interested in extending their Scala applications to work with Einstein Vision APIs.
You can find the code repo for the library and samples at https://github.com/rajdeepd/einstein-scala-lib . You can also try Einstein Vision on Trailhead to learn how to integrate Einstein Vision into your Force.com workflows using Apex and Visualforce. If you have any questions, feel free to reach out on the Salesforce Developer Forums.
About the author
Rajdeep Dua is a director of developer relations at Salesforce. He is passionate about helping developers learn about cloud computing, machine learning, and Salesforce. He has over 17 years of experience in software product development and developer relations.
Related links
https://metamind.io/
https://trailhead.salesforce.com/en/projects/predictive_vision_apex
https://github.com/rajdeepd/einstein-go-apis
https://github.com/rajdeepd/einstein-scala-lib