Using Einstein Vision Within Scala

Einstein Vision helps you build smarter applications by using deep learning to automatically recognize images. It provides an API that lets you use image reco

gnition to build AI-enabled apps. Today we’ll explore how to access Einstein Vision from Scala, which is a scalable language program.

Scala is a language that can be used on shell with one-line commands, or it can be used to build complex mission-critical languages. It’s becoming popular with programmers who want a robust language and libraries but don’t want to give up the convenience of a Java virtual machine (JVM).

Some features of Scala Language include:

  • Functional programming language
  • Object-oriented features
  • JVM-compliant

Because visual data is expanding rapidly at a 9% CAGR (compound annual growth rate), it’s vital for organizations to tap into nontextual data to discover conversations about their brand.

Some use-cases to consider include:

  • Tracking your brand perception on social channels like Twitter and Facebook
  • Giving field service agents the ability to identify and classify objects, such as factory equipment, to handle service updates faster

We cover details about algorithms used in Einstein Vision service in a separate blog post, Using Einstein Within Golang.

How does Einstein Vision work?

Einstein Vision is part of a supervised learning technique in which a model is trained on pre-labeled training data. The training dataset consists of labeled images and is loaded into Einstein via an API call. Next, a REST API call is made to train the dataset, and the output is a trained model, which has a unique model ID. An API call is made to predict the label of a new image that the model has never seen before. You can also find out metrics on accuracy of the model. This diagram shows the schematics of working with Einstein Vision service.

Scala Einstein library

In this section, you learn about the class structure and various functions exposed by the Scala library that can help you access the Vision API REST endpoints.

The class structure

The library has been divided into multiple packages with the base package being com.einstein.lib.

The main vision service methods are in com.einstein.lib.VisionService and com.einstein,lib.DataSetService.

Complete class hierarchy:

.
`— com
`— einstein
`— lib
|— Constants.scala
|— dataset
| |— CreateDataSetResponse.scala
| |— DataSet.scala
| |— DataSets.scala
| |— Label.scala
| `— LabelSummary.scala
|— example
| |— DataSetAndModelList.scala
| |— DataSetCreate.scala
| |— DataSetDelete.scala
| |— DataSetList.scala
| |— PredictLocalImage.scala
| |— PredictRemoteImage.scala
| `— ShowModelMetrics.scala
|— ImageToBase64.scala
|— model
| |— MetricsData.java
| |— Metrics.java
| |— Model.java
| `— Models.java
|— service
| |— DataSetService.scala
| |— PredictionResponse.scala
| |— Probabilities.scala
| `— VisionService.scala
|— types.scala
`— Util.scala

Vision service

This service trait is used to predict the label of an image residing in a local path or remote url:

trait VisionService {
  def predictImage(url: String, modelId: String, accessToken: String):
    java.util.List[_]
  def predictLocalImage(localFilePath: String, modelId: String, accessToken: String):
    List[_]
}

Dataset service

This service trait (interface in Scala) provides life cycle methods of the dataset and model:

trait DataSetService {
  def getDataSetsList(accessToken: String): List[_]
  def getDataSetAndModelList(accessToken: String): List[_]
  def getAllModels(accessToken: String, id: Long): List[_]
  def getModelMetrics(accessToken : String, id: String) : MetricsData
  def createDataSet(accessToken : String, path : String):
    CreateDataSetResponse
  def createDataSetFromLocalFile(accessToken : String, path : String):
    CreateDataSetResponse
  def deleteDataSet(accessToken: String, id : String) : String
  def trainDataSet(accessToken: String, id: String) : TrainDataSetResponse
}

Code examples

This section explores how dataset is created, used for training, and deleted.

Create a new dataset

Create a new dataset by providing the url of the zipped dataset folder with labels and images.

The following method createDataSet is defined in DataSetServiceImpl class:

def createDataSet(accessToken : String, path : String): CreateDataSetResponse = {
    val client: HttpClient = Util.getClient
    val url = "https://api.metamind.io/v1/vision/datasets/upload/sync"
    val post: HttpPost = new HttpPost(url)
    post.setHeader("Cache-Control", "no-cache")
    post.setHeader("Authorization", "Bearer " + accessToken)
    val builder: MultipartEntityBuilder = MultipartEntityBuilder.create
    builder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE)
    builder.addTextBody("path", path)
    val entity: HttpEntity = builder.build
    post.setEntity(entity)
    val handler: BasicResponseHandler = new BasicResponseHandler
    val response: HttpResponse = client.execute(post)
    val body: String = handler.handleResponse(response)
    val gson: Gson = new Gson
    val res: CreateDataSetResponse = gson.fromJson(body, classOf[CreateDataSetResponse])
    println(res)
    return res
}

Let’s look at an example code that uses this method. As you can see we are creating a new dataset from a zip file located at a URL by calling the the method createDataSet(..)

package com.einstein.lib.example

import java.util
import com.einstein.lib.Constants
import com.einstein.lib.dataset.{CreateDataSetResponse, DataSet}
import com.einstein.lib.model.MetricsData
import com.einstein.lib.service.DataSetServiceImpl

object DataSetCreate {
  def main(args: Array[String]) = {
    val accessToken = Constants.ACCESS_TOKEN
    val path = "http://metamind.io/images/mountainvsbeach.zip"
    val dataSetService = new DataSetServiceImpl
    val response = dataSetService.createDataSet(accessToken, path)
    println("id: " + response.getId)
    println("name: " + response.getName)
    println("created at: " + response.getCreatedAt)
  }
}

Train a dataset

Create a new dataset by providing the url of the zipped dataset folder with labels and images.

The following method trainDataSet is defined in DataSetServiceImpl class. This method takes accessToken and datasetId and the parameters.

def trainDataSet(accessToken: String, id: String) : TrainDataSetResponse = {
    val client: HttpClient = Util.getClient
    val url = Constants.BASE + "train" 
    val post: HttpPost = new HttpPost(url)
    post.setHeader("Cache-Control", "no-cache")
    post.setHeader("Authorization", "Bearer " + accessToken)
    val builder: MultipartEntityBuilder = MultipartEntityBuilder.create
    builder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE)
    builder.addTextBody("name", "Beach Mountain Model Test")
    builder.addTextBody("datasetId", id)
    val entity: HttpEntity = builder.build
    post.setEntity(entity)
    val handler: BasicResponseHandler = new BasicResponseHandler
    val response: HttpResponse = client.execute(post)
    println(response)
    val body: String = handler.handleResponse(response)
    val gson: Gson = new Gson
    val res: TrainDataSetResponse = gson.fromJson(body, classOf[TrainDataSetResponse])
    return res
  }
}

Create a new dataset by providing the url of the zipped dataset folder with labels and images.

The following method trainDataSet is defined in DataSetServiceImpl class:

package com.einstein.lib.example

import com.einstein.lib.Constants
import com.einstein.lib.service.DataSetServiceImpl

object DataSetTrain {
  def main(args: Array[String]) = {
    val accessToken = Constants.ACCESS_TOKEN
    val datasetId = "1004151"
    val dataSetService = new DataSetServiceImpl
    val response = dataSetService.trainDataSet(accessToken, datasetId)
    println(response)
  }
}

1. Instantiate accessToken from environment variable.
2. Create datasetId variable with the appropriate value.
3. Instantiate DataSetServiceImpl.
4. Call dataSetService.trainDataSet(.., ..) with appropriate parameters.

Predict label of an image

Let’s look at how to predict an image using this library. We’re going to use two methods defined in VisionServiceImpl. Method predictImage() is used to predict remote image located at url. If the image is located in a local path, the library provides another method predictLocalImage.

def predictImage(url: String, modelId: String, accessToken: String):
java.util.List[_]

def predictLocalImage(localFilePath: String, modelId: String, accessToken: String):
List[_]

Implementation of both these methods can be found as below.

Predict remote image:

def predictImage(url: String, modelId: String, accessToken: String): java.util.List[_] = {
    val client: HttpClient = Util.getClient
    val post: HttpPost = new HttpPost(Constants.VISION_PREDICT_URL)
    post.setHeader("Cache-Control", "no-cache")
    post.setHeader("Authorization", "Bearer " + accessToken)
    val builder: MultipartEntityBuilder = MultipartEntityBuilder.create
    builder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE)
    builder.addTextBody("sampleLocation", url)
    builder.addTextBody("modelId", modelId)
    val entity: HttpEntity = builder.build
    post.setEntity(entity)
    val handler: BasicResponseHandler = new BasicResponseHandler
    val response: HttpResponse = client.execute(post)
    val body: String = handler.handleResponse(response)
    val gson: Gson = new Gson
    val res: PredictionResponse = gson.fromJson(body, 
      classOf[PredictionResponse]).asInstanceOf[PredictionResponse]
    val probabilitiesList: java.util.List[_] = res.getProbabilities
    return probabilitiesList
  }

Code walkthrough for predictImage() method:

1. Create org.apache.http.client.HttpClientinstance, we are using Apache’s HTTP library.
2. Create org.apache.http.client.methods.HttpPost Object using Vision Prediction url
3. Add HTTP headers Cache-Control and OAuth access token using Authorization key.
4. Add HTTP body using org.apache.http.entity.mime.MultipartEntityBuilder.
5. This body contains key, values for sampleLocation and modelId.
6. Create a HttpResponseHandler using BasicResponseHandler instance.
7. Execute the client.
8. Extract the body from HttpResponse object and parse JSON in the body using GSON library.
9. Extract the List of probabilities and return the method call with the List object.

Predict local image:

def predictLocalImage(localFilePath: String, modelId: String, accessToken: String):List[_] = {
    val client: HttpClient = Util.getClient
    val post: HttpPost = new HttpPost(Constants.VISION_PREDICT_URL)
    post.setHeader("Cache-Control", "no-cache")
    post.setHeader("Authorization", "Bearer " + accessToken)
    val builder: MultipartEntityBuilder = MultipartEntityBuilder.create
    builder.setMode(HttpMultipartMode.BROWSER_COMPATIBLE)
    val imageDataString: String = ImageToBase64.encode(localFilePath)
    builder.addTextBody("sampleBase64Content", imageDataString)
    builder.addTextBody("modelId", modelId)
    val entity: HttpEntity = builder.build
    post.setEntity(entity)
    val handler: BasicResponseHandler = new BasicResponseHandler
    val response: HttpResponse = client.execute(post)
    val body: String = handler.handleResponse(response)
    val gson: Gson = new Gson
    val res: PredictionResponse = gson.fromJson(body, classOf[PredictionResponse]).asInstanceOf[PredictionResponse]
    val probabilitiesList: java.util.List[_] = res.getProbabilities
    return probabilitiesList
  }

Using the predict image APIs

Local image prediction example

In the next example, a remote image specified by localfilePath is classified using a preexisting model,modelId=FoodImageClassifier

package com.einstein.lib.example

import java.util

import com.einstein.lib.service.{VisionService, Probabilities}
import com.einstein.lib.Constants
import com.einstein.lib.service.VisionServiceImpl

object PredictLocalImage {
  def main(args: Array[String]) = {
    val url = "http://metamind.io/images/foodimage.jpg"
    val localFilePath = "/home/ubuntu/Downloads/samosa.jpeg"
    val modelId = "FoodImageClassifier"
    val accessToken = Constants.ACCESS_TOKEN
    val visionService = new VisionServiceImpl()
    val list = visionService.predictLocalImage(localFilePath, modelId, accessToken)
    val itr = list.iterator()
    println( "Label : Probability")
    while(itr.hasNext){
      val probabilities =  itr.next().asInstanceOf[Probabilities]
      val label = probabilities.getLabel
      val probability = probabilities.getProbability
      println(label + " : " + probability)
    }
  }
}

Code walkthrough:

1. In the main method of the object PerdictLocalImage define variable localFilePath for the local path of image file for which label needs to be predicted.
2. Instantiate variable modelId model to be used, FoodImageClassifier in this case.
3. Instantiate accessToken variable, defined in the environment key access_token
4. Instantiate VisionServiceImpl object visionService.
5. Call predictImage(...) method on the visionService object.
6. Iterate over returned list of probabilities and print label and corresponding probability.

Label for the food image, samosa, in this case, being predicted is shown below:

Output of the program listing is:

Label : Probability
samosa : 0.96255666
wonton : 0.026478248
spanakopita : 0.0061982954
gyoza : 0.0018568092
dumpling : 0.0014262322

Remote image prediction example

In the next example, a remote image specified by a url is classified using a preexisting model, modelId=FoodImageClassifier.

package com.einstein.lib.example
import com.einstein.lib.service.{VisionServiceImpl, VisionService, Probabilities}
import com.einstein.lib.Constants

object PredictRemoteImage {
  def main(args: Array[String]) = {
    val url = "https://upload.wikimedia.org/wikipedia/commons/thumb/d/d1/Pepperoni_pizza.jpg/440px-Pepperoni_pizza.jpg"
    val modelId = "FoodImageClassifier"
    val accessToken = Constants.ACCESS_TOKEN
    val visionService = new VisionServiceImpl()
    val list = visionService.predictImage(url,modelId, accessToken)
    val itr = list.iterator()
    println( "Label : Probability")
    while(itr.hasNext){
      val probabilities =  itr.next().asInstanceOf[Probabilities]
      val label = probabilities.getLabel
      val probability = probabilities.getProbability
      println(label + " : " + probability)
    }
  }
}

Code walkthrough:

1. In the main method of the object define PerdictRemoteImage variable for url the image to be predicted.
2. Instantiate variable modelId model to be used, FoodImageClassifier in this case.
3. Instantiate variable accessToken, Instantiate object defined in the environment variable access_token.
4. VisionServiceImpl visionService.
5. Call predictImage(...) method on the visionService object.
6. Iterate over returned list of probabilities and print label and corresponding probability.

Food image being predicted, pizza, in this case, is shown below:

 

The output of the listing for remote image prediction:

Label : Probability
pizza : 0.5730966
pepperoni : 0.3060464
meatball : 0.030234689
baked_ziti : 0.013054437
focaccia : 0.011084515

Compiling and running the samples

In the library sbt tool is being used for compiling the samples. Please review the build.sbt file below which lists the library dependencies

name := "einstein-scala-lib"

version := "1.0"

scalaVersion := "2.11.6"

libraryDependencies += "org.apache.httpcomponents" % "httpclient" % "4.5.3"
libraryDependencies += "org.apache.httpcomponents" % "httpcore" % "4.4.6"
libraryDependencies += "com.google.code.gson" % "gson" % "1.7.1"
libraryDependencies += "org.apache.httpcomponents" % "httpmime" % "4.5.3"
libraryDependencies += "org.apache.commons" % "commons-io" % "1.3.2"
libraryDependencies += "commons-lang" % "commons-lang" % "2.6"

You can compile and run the samples using the following commands, please note PredictLocalImage is just an example

sbt compile
sbt "run-main com.einstein.lib.example.PredictLocalImage"

Summary

This blog post provides you with the basics on how to use Einstein Vision from a Scala stand-alone application. Hopefully, these samples provide a good starting point for developers interested in extending their Scala applications to work with Einstein Vision APIs.

You can find the code repo for the library and samples at https://github.com/rajdeepd/einstein-scala-lib . You can also try Einstein Vision on Trailhead to learn how to integrate Einstein Vision into your Force.com workflows using Apex and Visualforce. If you have any questions, feel free to reach out on the Salesforce Developer Forums.

About the author

Rajdeep Dua is a director of developer relations at Salesforce. He is passionate about helping developers learn about cloud computing, machine learning, and Salesforce. He has over 17 years of experience in software product development and developer relations.

Related links
https://metamind.io/
https://trailhead.salesforce.com/en/projects/predictive_vision_apex
https://github.com/rajdeepd/einstein-go-apis
https://github.com/rajdeepd/einstein-scala-lib

Leave your comments...

Using Einstein Vision Within Scala