Einstein Platform Services supports Einstein Vision and Einstein Language APIs. These APIs help you build smarter applications by using deep learning to automatically recognize images and detect the sentiment and intent of text with image recognition technology and natural language processing (NLP). In this post, we’ll explore how the Einstein Platform Services APIs can be accessed from a Python application to help you build smarter applications faster.
Before we get into the details, let’s first take a look at the algorithms that power Einstein Vision and Einstein Language APIs.
The Algorithms Powering Einstein Vision and Einstein Language
As we’ve learned in previous blogs, deep learning based neural networks are used for text classification and image classification. The topology of neural networks can differ, for example, text classification labels can be intent based or sentiment based. Discussing neural networks in great detail is outside the scope of this blog, but if you want to get smarter about the fundamentals of AI, read this playbook.
It’s outside the scope of this blog to discuss neural networks in great detail, but feel free to start with the information available online. In general, think of them as multiple neurons tied together in a graph.
Einstein Vision APIs include an input layer where images are fed and a hidden layer, which extracts each feature of the image’s contours, corners, and shapes to predict new images.
In regard to Einstein Language APIs, concepts like word2vector, embedding, and parts of speech are used in association with different kinds of neural network topologies to determine weights and biases for neural nets.
As we learned in this Einstein Vision blog, the last layer in a neural network is the output layer, which predicts output labels. This same output layer works for Einstein Vision and Einstein Language and depending on the dataset, a different neural network topology is chosen which affects prediction and classification results.
How do Einstein Vision and Einstein Language work?
Einstein Platform Services use supervised learning techniques to train models on labeled training data. The training dataset consists of labeled images or labeled text documents and is uploaded to Einstein Platform Services via an API call. Next, a REST API call is made to train the dataset and the output is a trained model with a unique model ID. Predicting classifiers for new images that the model has never seen before is also executed through an API call.
Python Einstein Library
In this section, you will learn about the class structure and various functions exposed by the Python Library. You can also refer to the Python QuickStart on the Einstein docs site.
The class structure
Python Wrapper
Wrappers are contained in a few classes and divided into global packages einstein and sub packages of language or vision.
|— einstein | |— constants.py | | | |— __init__.py | | | |— language | | |— dataset.py | | |— __init__.py | | |— prediction.py | | | |— dataset.py | |— __init__.py | |— prediction.py | `— requirements.txt
Requests Python Library
We are going to use the requests Python Library to make the HTTP GET, POST, and DELETE calls to the remote Einstein endpoints.
Library Internals
HTTP GET Request Implementation
The code listing below shows how a GET request is made Einstein endpoint. Parameters needed are a multipart object with appropriate fields and the URL to get called.
import requests
from requests_toolbelt.multipart.encoder import MultipartEncoder
from einstein.constants import DATASETS_URL
multipart_data = MultipartEncoder( fields={'type': 'image'})
headers = {'Authorization': 'Bearer ' + self.access_token,'Content-Type': multipart_data.content_type}
res = requests.get(DATASETS_URL + '/' + id, headers=headers, data=multipart_data)
HTTP POST request implementation
The code listing below shows how a POST request is made to Einstein endpoint. Parameters needed are a MultipartEncoder object with appropriate fields and the URL to get called.
import requests
from requests_toolbelt.multipart.encoder import MultipartEncoder
from einstein.constants import DATASETS_URL
multipart_data = MultipartEncoder(
fields={'path': path,'type': 'image'})
headers = {'Authorization': 'Bearer ' + self.access_token,
'Content-Type': multipart_data.content_type}
res = requests.post(DATASETS_URL + '/upload',headers=headers, data=multipart_data)
Einstein Vision Code examples
Create a DataSet
First step in using Einstein APIs is to upload a DataSet from a local directory or a remote URL. In the example below we are loading the dataset from a zip file.
import json
from einstein.vision.dataset import DataSet
from einstein.constants import ACCESS_TOKEN
from einstein.constants import TYPE_IMAGE
def main():
access_token = ACCESS_TOKEN
dataset = DataSet(access_token=access_token)
path = 'http://metamind.io/images/mountainvsbeach.zip'
response = dataset.create_dataset(path, TYPE_IMAGE)
print(json.dumps(response, indent=4, sort_keys=True))
if __name__ == "__main__":
main()
create_dataset(..) function is implemented in DataSet class and is explained below. It internally calls _create_dataset(..) with appropriate image classification type.
class DataSet:
def __init__(self,access_token):
self.access_token = access_token
def create_dataset(self, path, y):
type = "image"
return self._create_dataset(path, type)
def _create_dataset(self, path, type):
multipart_data = MultipartEncoder(
fields={
'path': path,
'type': type
}
)
headers = {'Authorization': 'Bearer ' + self.access_token,
'Content-Type': multipart_data.content_type}
res = requests.post(DATASETS_URL + '/upload',
headers=headers, data=multipart_data)
json_response = json.loads(res.text)
return json_response
Code walkthrough
1. Instantiate DataSet object by providing an access_token.
2. Call create_dataset() function in the DataSet object with path variableurl, this internally calls function_create_dataset(..).
3. Create MultipartEncoder object with fields for path and type.
4. Create headers
headers = {'Authorization': 'Bearer ' + self.access_token,
'Content-Type': multipart_data.content_type}
5. Call requests.post() where the remote call happens
function res = requests.post(DATASETS_URL + '/upload',
headers=headers,
data=multipart_data)
6. Convert response body to JSON and return using json.loads(res.text) where res is the response from the remote call.
After the dataset has been created, the next step is to train a model based on this dataset.
Train a Vision model
Once the DataSet has been created next step is to train a model. Listing below shows how a vision model can be trained from the DataSet id created in the previous section.
import json
from einstein.vision.dataset import DataSet
from einstein.constants import ACCESS_TOKEN
def main():
access_token = ACCESS_TOKEN
id = '1010488'
dataset = DataSet(access_token=access_token)
response = dataset.train_dataset(id)
if('available' in response):
print(json.dumps(response, indent=4, sort_keys=True))
else:
print('Response status ok?: ' + str(response.ok))
print(json.dumps(response.text, indent=4, sort_keys=True))
if __name__ == "__main__":
main()
class DataSet implements the train_dataset(..) function. id of the dataset is passed to this function.
def train_dataset(self, id):
multipart_data = MultipartEncoder(
fields={'name': 'Beach Mountain Model Test',
'datasetId' : id})
headers = {'Authorization': 'Bearer ' + self.access_token,
'Content-Type': multipart_data.content_type}
res = requests.post( VISION_BASE_URL + '/train',
headers=headers, data=multipart_data)
return res
Code walkthrough
1. Instantiate DataSet object by providing an access_token
2. Call tain_dataset(self,id) function in DataSet object with the dataset id . This id is 1010488 in the example listed above.
3. Create MultipartEncoder object with fields for path and type
4. Create headers json. Both MultipartEncoder and header objects are passed to the requests.post(..) call
headers = {'Authorization' : 'Bearer ' + self.access_token,
datasetId' : id})
5. Call requests.post(..) function
res = requests.post( VISION_BASE_URL + '/train',
headers=headers, data=multipart_data)
Predict the label of an Image
Once the model is trained the model id is returned, it can be used to predict label of an image. Listing below shows how this prediction is made for a remote image.
import json
from einstein.vision.prediction import Prediction
from einstein.constants import ACCESS_TOKEN
def main():
access_token = ACCESS_TOKEN
url = 'https://upload.wikimedia.org/wikipedia/commons/thumb/d/d1/Pepperoni_pizza.jpg/'
'440px-Pepperoni_pizza.jpg'
model_id = 'FoodImageClassifier'
prediction = Prediction(access_token=access_token)
response = prediction.predict_remote_image(url, model_id)
probabilities = response['probabilities']
for x in probabilities:
print(str(x['label']) + ":" + str(x['probability']))
if __name__ == "__main__":
main()
Where class Prediction‘s method predict_remote_image(..) does the heavy lifting of making the HTTP POST call
class Prediction:
def __init__(self,access_token):
self.access_token = access_token
def predict_remote_image(self, url, model_id):
multipart_data = MultipartEncoder(
fields={'sampleLocation': url,
'modelId' : model_id})
headers = {'Authorization': 'Bearer ' + self.access_token,
'Content-Type': multipart_data.content_type}
res = requests.post(VISION_PREDICT_URL,
headers=headers, data=multipart_data)
if res.ok:
json_response = json.loads(res.text)
return json_response
else:
return res
Output
When we run the program listed above, it gives output below. We used the out-of-box classifier (FoodImageClassifier), and it was able to predict that this image is a pizza and also do a more fine-grained classification between pizza and pepperoni.
pizza:0.5730966 pepperoni:0.306046 meatball:0.030234689 baked_ziti:0.013054437 focaccia:0.011084515
In the next section, we look at Einstein Language APIs usage from Python
Einstein Language Service examples
In this section, we look at how Einstein’s Language APIs can be used to predict an intent or a sentiment.
DataSet Creation
Here we are using the sample data file available on the Einstein documentation website.
"what's the weather look like",current-weather "is it raining",current-weather "what's the temperature",current-weather "show me current conditions",current-weather "how hot is it",current-weather "how cold is it",current-weather "is it snowing",current-weather ....
We have a different DataSet class for Intent and Sentiment APIs in the package einstein.language.
import json
from einstein.language.dataset import DataSet
from einstein.constants import ACCESS_TOKEN
def main():
access_token = ACCESS_TOKEN
dataset = DataSet(access_token=access_token)
path = 'http://einstein.ai/text/weather.csv'
response = dataset.create_intent_dataset(path)
print(json.dumps(response, indent=4, sort_keys=True))
if __name__ == "__main__":
main()
Where listing for create_intent_dataset(..) is given below.
def create_intent_dataset(self, path): type = 'text-intent' return self._create_dataset(path, type)
This function in-turn calls _create_dataset(..)
def _create_dataset(self, path, type):
multipart_data = MultipartEncoder(
fields={
'path': path,
'type': type
}
)
headers = {'Authorization': 'Bearer ' + self.access_token,
'Content-Type': multipart_data.content_type}
res = requests.post(LANG_DATASETS_URL + '/upload',
headers=headers, data=multipart_data)
json_response = json.loads(res.text)
return json_response
The HTTP POST request is made to the URL https://api.einstein.ai/v2/language/datasets/upload
Where constant LANG_DATASETS_URL is defined as show below:
LANG_BASE_URL = EINSTEIN_BASE_URL + '/language' LANG_DATASETS_URL = LANG_BASE_URL + '/datasets'
Code Walkthrough
1. First we instantiate MultipartEncoder class, add path and type values. type is text-intent in this case.
2. Instantiate headers json with Authorization and Content-Type
3. Call requests.post(...) to make the create call
4. Parse the response json and load it into json_response variable
Train the DataSet
Once the weather dataset is created we can train it using the Dataset class. Listing below shows how the DataSet is trained.
import json
from einstein.language.dataset import DataSet
from einstein.constants import ACCESS_TOKEN
def main():
access_token = ACCESS_TOKEN
id = '1010879'
name = 'Weather Intent Model'
dataset = DataSet(access_token=access_token)
response = dataset.train_dataset(id, model_name=name)
if('available' in response):
print(json.dumps(response, indent=4, sort_keys=True))
else:
print('Response status ok?: ' + str(response.ok))
print(json.dumps(response.text, indent=4, sort_keys=True))
if __name__ == "__main__":
main()
Code Walkthrough
1. First we get the ACCESS_TOKEN
2. Assign a name to the model
3. Instantiate DataSet class with access_token in the constructor
4. Call dataset.train_dataset(..)
5. Extract json from the response and print
Predict an Intent
Once the model is ready we use it to predict the intent by giving the document and the model id.
import json
from einstein.language.prediction import Prediction
from einstein.constants import ACCESS_TOKEN
def main():
access_token = ACCESS_TOKEN
document = 'what is the weather in los angeles'
model_id = 'V3E7CDCNLYMQD7XFM3NBXMZICE'
prediction = Prediction(access_token=access_token)
response = prediction.predict_intent(document, model_id)
try:
probabilities = response['probabilities']
for x in probabilities:
print(str(x['label']) + ":" + str(x['probability']))
except TypeError :
print('response ok? ' + str(response.ok))
print('response content: ' + str(response.content))
return True
if __name__ == "__main__":
main()
Code Walkthrough
1. Get the access_token which is defined in einstein.Constants file
2. Initialize document and model_id
3. Initialize Prediction class
4. Call prediction.predict_intent(..)
5. Parse the response and list the probabilities
Output
On executing the code above an output is obtained which shows the intent type and its probability. Intent type is custom intent which is given as part of the training dataset. Higher the probability value, closer the input phrase is to the intent.
current-weather:0.9848461 five-day-forecast:0.009139863 hourly-forecast:0.0060140914
Summary
This blog post provides you with the basics on how to use Einstein Vision and Language APIs from a Python stand-alone application. Hopefully, these samples provide a good starting point for developers interested in extending their python applications to work with Einstein Vision and Language APIs. You can find the code repo for the library and samples here.
You can also try a project on Einstein Vision on Trailhead to learn how to integrate Einstein Vision into your Salesforce Platform workflows using Apex and Visualforce. If you have any questions, feel free to reach out on our forums.
About the author
Rajdeep Dua is a Director of Developer Relations at Salesforce. He is passionate about helping developers learn about cloud computing, machine learning, and Salesforce platform. He has over 18 years of experience in software product development and developer relations.