Einstein Platform Services supports Einstein Vision and Einstein Language APIs. These APIs help you build smarter applications by using deep learning to automatically recognize images and detect the sentiment and intent of text with image recognition technology and natural language processing (NLP). In this post, we’ll explore how the Einstein Platform Services APIs can be accessed from a Python application to help you build smarter applications faster.
Before we get into the details, let’s first take a look at the algorithms that power Einstein Vision and Einstein Language APIs.
The Algorithms Powering Einstein Vision and Einstein Language
As we’ve learned in previous blogs, deep learning based neural networks are used for text classification and image classification. The topology of neural networks can differ, for example, text classification labels can be intent based or sentiment based. Discussing neural networks in great detail is outside the scope of this blog, but if you want to get smarter about the fundamentals of AI, read this playbook.
It’s outside the scope of this blog to discuss neural networks in great detail, but feel free to start with the information available online. In general, think of them as multiple neurons tied together in a graph.
Einstein Vision APIs include an input layer where images are fed and a hidden layer, which extracts each feature of the image’s contours, corners, and shapes to predict new images.
In regard to Einstein Language APIs, concepts like word2vector, embedding, and parts of speech are used in association with different kinds of neural network topologies to determine weights and biases for neural nets.
As we learned in this Einstein Vision blog, the last layer in a neural network is the output layer, which predicts output labels. This same output layer works for Einstein Vision and Einstein Language and depending on the dataset, a different neural network topology is chosen which affects prediction and classification results.
How do Einstein Vision and Einstein Language work?
Einstein Platform Services use supervised learning techniques to train models on labeled training data. The training dataset consists of labeled images or labeled text documents and is uploaded to Einstein Platform Services via an API call. Next, a REST API call is made to train the dataset and the output is a trained model with a unique model ID. Predicting classifiers for new images that the model has never seen before is also executed through an API call.
Python Einstein Library
In this section, you will learn about the class structure and various functions exposed by the Python Library. You can also refer to the Python QuickStart on the Einstein docs site.
The class structure
Wrappers are contained in a few classes and divided into global packages
einstein and sub packages of
Requests Python Library
We are going to use the requests Python Library to make the HTTP GET, POST, and DELETE calls to the remote Einstein endpoints.
HTTP GET Request Implementation
The code listing below shows how a GET request is made Einstein endpoint. Parameters needed are a multipart object with appropriate fields and the URL to get called.
HTTP POST request implementation
The code listing below shows how a POST request is made to Einstein endpoint. Parameters needed are a MultipartEncoder object with appropriate fields and the URL to get called.
Einstein Vision Code examples
Create a DataSet
First step in using Einstein APIs is to upload a DataSet from a local directory or a remote URL. In the example below we are loading the dataset from a zip file.
create_dataset(..) function is implemented in DataSet class and is explained below. It internally calls _create_dataset(..) with appropriate image classification type.
DataSet object by providing an
create_dataset() function in the
DataSet object with path variable
url, this internally calls function
MultipartEncoder object with fields for
path and type.
4. Create headers
requests.post() where the remote call happens
6. Convert response body to JSON and return using
res is the response from the remote call.
After the dataset has been created, the next step is to train a model based on this dataset.
Train a Vision model
Once the DataSet has been created next step is to train a model. Listing below shows how a vision model can be trained from the DataSet id created in the previous section.
class DataSet implements the train_dataset(..) function. id of the dataset is passed to this function.
DataSet object by providing an
tain_dataset(self,id) function in
DataSet object with the dataset
id . This id is
1010488 in the example listed above.
MultipartEncoder object with fields for
4. Create headers json. Both MultipartEncoder and header objects are passed to the requests.post(..) call
Predict the label of an Image
Once the model is trained the model id is returned, it can be used to predict label of an image. Listing below shows how this prediction is made for a remote image.
predict_remote_image(..) does the heavy lifting of making the HTTP POST call
When we run the program listed above, it gives output below. We used the out-of-box classifier (
FoodImageClassifier), and it was able to predict that this image is a pizza and also do a more fine-grained classification between pizza and pepperoni.
In the next section, we look at Einstein Language APIs usage from Python
Einstein Language Service examples
In this section, we look at how Einstein’s Language APIs can be used to predict an intent or a sentiment.
Here we are using the sample data file available on the Einstein documentation website.
We have a different
DataSet class for Intent and Sentiment APIs in the package
Where listing for
create_intent_dataset(..) is given below.
This function in-turn calls _create_dataset(..)
The HTTP POST request is made to the URL
LANG_DATASETS_URL is defined as show below:
1. First we instantiate
MultipartEncoder class, add
text-intent in this case.
2. Instantiate headers json with Authorization and Content-Type
requests.post(...) to make the create call
4. Parse the response json and load it into
Train the DataSet
Once the weather dataset is created we can train it using the Dataset class. Listing below shows how the DataSet is trained.
1. First we get the
2. Assign a name to the model
DataSet class with
access_token in the constructor
json from the response and print
Predict an Intent
Once the model is ready we use it to predict the intent by giving the document and the model id.
1. Get the
access_token which is defined in einstein.Constants file
5. Parse the
response and list the probabilities
On executing the code above an output is obtained which shows the intent type and its probability. Intent type is custom intent which is given as part of the training dataset. Higher the probability value, closer the input phrase is to the intent.
This blog post provides you with the basics on how to use Einstein Vision and Language APIs from a Python stand-alone application. Hopefully, these samples provide a good starting point for developers interested in extending their python applications to work with Einstein Vision and Language APIs. You can find the code repo for the library and samples here.
You can also try a project on Einstein Vision on Trailhead to learn how to integrate Einstein Vision into your Salesforce Platform workflows using Apex and Visualforce. If you have any questions, feel free to reach out on our forums.
About the author
Rajdeep Dua is a Director of Developer Relations at Salesforce. He is passionate about helping developers learn about cloud computing, machine learning, and Salesforce platform. He has over 18 years of experience in software product development and developer relations.