Using Einstein APIs Within Python

Einstein Platform Services supports Einstein Vision and Einstein Language APIs. These APIs help you build smarter applications by using deep learning to automatically recognize images and detect the sentiment and intent of text with image recognition technology and natural language processing (NLP). In this post, we’ll explore how the Einstein Platform Services APIs can be accessed from a Python application to help you build smarter applications faster.

Before we get into the details, let’s first take a look at the algorithms that power Einstein Vision and Einstein Language APIs.

The Algorithms Powering Einstein Vision and Einstein Language

As we’ve learned in previous blogs, deep learning based neural networks are used for text classification and image classification. The topology of neural networks can differ, for example, text classification labels can be intent based or sentiment based. Discussing neural networks in great detail is outside the scope of this blog, but if you want to get smarter about the fundamentals of AI, read this playbook.

It’s outside the scope of this blog to discuss neural networks in great detail, but feel free to start with the information available online. In general, think of them as multiple neurons tied together in a graph.
Einstein Vision APIs include an input layer where images are fed and a hidden layer, which extracts each feature of the image’s contours, corners, and shapes to predict new images.

In regard to Einstein Language APIs, concepts like word2vector, embedding, and parts of speech are used in association with different kinds of neural network topologies to determine weights and biases for neural nets.

As we learned in this Einstein Vision blog, the last layer in a neural network is the output layer, which predicts output labels. This same output layer works for Einstein Vision and Einstein Language and depending on the dataset, a different neural network topology is chosen which affects prediction and classification results.

How do Einstein Vision and Einstein Language work?

Einstein Platform Services use supervised learning techniques to train models on labeled training data. The training dataset consists of labeled images or labeled text documents and is uploaded to Einstein Platform Services via an API call. Next, a REST API call is made to train the dataset and the output is a trained model with a unique model ID. Predicting classifiers for new images that the model has never seen before is also executed through an API call.

Python Einstein Library

In this section, you will learn about the class structure and various functions exposed by the Python Library. You can also refer to the Python QuickStart on the Einstein docs site.

The class structure

Python Wrapper
Wrappers are contained in a few classes and divided into global packages einstein and sub packages of language or vision.

|— einstein
| |— constants.py
| |
| |— __init__.py
| |
| |— language
| | |— dataset.py
| | |— __init__.py
| | |— prediction.py
| | 
| |— dataset.py
| |— __init__.py
| |— prediction.py
|
`— requirements.txt

Requests Python Library

We are going to use the requests Python Library to make the HTTP GET, POST, and DELETE calls to the remote Einstein endpoints.

Library Internals

HTTP GET Request Implementation

The code listing below shows how a GET request is made Einstein endpoint. Parameters needed are a multipart object with appropriate fields and the URL to get called.

import requests
from requests_toolbelt.multipart.encoder import MultipartEncoder
from einstein.constants import DATASETS_URL

multipart_data = MultipartEncoder( fields={'type': 'image'})
headers = {'Authorization': 'Bearer ' + self.access_token,'Content-Type': multipart_data.content_type}
res = requests.get(DATASETS_URL + '/' + id, headers=headers, data=multipart_data)

HTTP POST request implementation

The code listing below shows how a POST request is made to Einstein endpoint. Parameters needed are a MultipartEncoder object with appropriate fields and the URL to get called.

import requests
from requests_toolbelt.multipart.encoder import MultipartEncoder
from einstein.constants import DATASETS_URL

multipart_data = MultipartEncoder(
fields={'path': path,'type': 'image'})
headers = {'Authorization': 'Bearer ' + self.access_token,
           'Content-Type': multipart_data.content_type}
res = requests.post(DATASETS_URL + '/upload',headers=headers, data=multipart_data)

Einstein Vision Code examples

Create a DataSet

First step in using Einstein APIs is to upload a DataSet from a local directory or a remote URL. In the example below we are loading the dataset from a zip file.

import json

from einstein.vision.dataset import DataSet
from einstein.constants import ACCESS_TOKEN
from einstein.constants import TYPE_IMAGE

def main():
    access_token = ACCESS_TOKEN
    dataset = DataSet(access_token=access_token)
    path = 'http://metamind.io/images/mountainvsbeach.zip'
    response = dataset.create_dataset(path, TYPE_IMAGE)
    print(json.dumps(response, indent=4, sort_keys=True))

if __name__ == "__main__":
    main()

create_dataset(..) function is implemented in DataSet class and is explained below. It internally calls _create_dataset(..) with appropriate image classification type.

class DataSet:
    def __init__(self,access_token):
        self.access_token = access_token

    def create_dataset(self, path, y):
        type = "image"
        return self._create_dataset(path, type)

    def _create_dataset(self, path, type):
        multipart_data = MultipartEncoder(
            fields={
                'path': path,
                'type': type
            }
        )
        headers = {'Authorization': 'Bearer ' + self.access_token,
                   'Content-Type': multipart_data.content_type}
        res = requests.post(DATASETS_URL + '/upload',
                            headers=headers, data=multipart_data)
        json_response = json.loads(res.text)
        return json_response

Code walkthrough

1. Instantiate DataSet object by providing an access_token.
2. Call create_dataset() function in the DataSet object with path variableurl, this internally calls function_create_dataset(..).
3. Create MultipartEncoder object with fields for path and type.
4. Create headers

headers = {'Authorization': 'Bearer ' + self.access_token,
             'Content-Type': multipart_data.content_type}

5. Call requests.post() where the remote call happens

function res = requests.post(DATASETS_URL + '/upload',
                               headers=headers, 
                               data=multipart_data)

6. Convert response body to JSON and return using json.loads(res.text) where res is the response from the remote call.

After the dataset has been created, the next step is to train a model based on this dataset.

Train a Vision model

Once the DataSet has been created next step is to train a model. Listing below shows how a vision model can be trained from the DataSet id created in the previous section.

import json
from einstein.vision.dataset import DataSet
from einstein.constants import ACCESS_TOKEN

def main():
    access_token = ACCESS_TOKEN
    id = '1010488'
    dataset = DataSet(access_token=access_token)
    response = dataset.train_dataset(id)
    if('available' in response):
        print(json.dumps(response, indent=4, sort_keys=True))
    else:
        print('Response status ok?: ' + str(response.ok))
        print(json.dumps(response.text, indent=4, sort_keys=True))


if __name__ == "__main__":
    main()

class DataSet implements the train_dataset(..) function. id of the dataset is passed to this function.

def train_dataset(self, id):
    multipart_data = MultipartEncoder(
        fields={'name': 'Beach Mountain Model Test',
                'datasetId' : id})

    headers = {'Authorization': 'Bearer ' + self.access_token,
               'Content-Type': multipart_data.content_type}
    res = requests.post( VISION_BASE_URL + '/train',
                          headers=headers, data=multipart_data)
    return res

Code walkthrough

1. Instantiate DataSet object by providing an access_token
2. Call tain_dataset(self,id) function in  DataSet object with the dataset id . This id is 1010488 in the example listed above.
3. Create MultipartEncoder object with fields for path and type
4. Create headers json. Both MultipartEncoder and header objects are passed to the requests.post(..) call

headers = {'Authorization' : 'Bearer ' + self.access_token,
              datasetId' : id})

5. Call requests.post(..) function

res = requests.post( VISION_BASE_URL + '/train',
                          headers=headers, data=multipart_data)

Predict the label of an Image

Once the model is trained the model id is returned, it can be used to predict label of an image. Listing below shows how this prediction is made for a remote image.

import json
from einstein.vision.prediction import Prediction
from einstein.constants import ACCESS_TOKEN

def main():
    access_token = ACCESS_TOKEN
    url = 'https://upload.wikimedia.org/wikipedia/commons/thumb/d/d1/Pepperoni_pizza.jpg/' 
          '440px-Pepperoni_pizza.jpg'
    model_id = 'FoodImageClassifier'
    prediction = Prediction(access_token=access_token)
    response = prediction.predict_remote_image(url, model_id)
    probabilities = response['probabilities']
    for x in probabilities:
        print(str(x['label']) + ":" + str(x['probability']))

if __name__ == "__main__":
    main()

Where class Prediction‘s method predict_remote_image(..) does the heavy lifting of making the HTTP POST call

class Prediction:

    def __init__(self,access_token):
        self.access_token = access_token

    def predict_remote_image(self, url, model_id):
        multipart_data = MultipartEncoder(
            fields={'sampleLocation': url,
                    'modelId' : model_id})

        headers = {'Authorization': 'Bearer ' + self.access_token,
                   'Content-Type': multipart_data.content_type}
        res = requests.post(VISION_PREDICT_URL,
                            headers=headers, data=multipart_data)
        if res.ok:
            json_response = json.loads(res.text)
            return json_response
        else:
            return res

Output
When we run the program listed above, it gives output below. We used the out-of-box classifier (FoodImageClassifier), and it was able to predict that this image is a pizza and also do a more fine-grained classification between pizza and pepperoni.

pizza:0.5730966
pepperoni:0.306046
meatball:0.030234689
baked_ziti:0.013054437
focaccia:0.011084515

In the next section, we look at Einstein Language APIs usage from Python

Einstein Language Service examples

In this section, we look at how Einstein’s Language APIs can be used to predict an intent or a sentiment.

DataSet Creation

Here we are using the sample data file available on the Einstein documentation website.

"what's the weather look like",current-weather
"is it raining",current-weather
"what's the temperature",current-weather
"show me current conditions",current-weather
"how hot is it",current-weather
"how cold is it",current-weather
"is it snowing",current-weather
....

We have a different DataSet class for Intent and Sentiment APIs in the package einstein.language.

import json
from einstein.language.dataset import DataSet
from einstein.constants import ACCESS_TOKEN

def main():
    access_token = ACCESS_TOKEN
    dataset = DataSet(access_token=access_token)
    path = 'http://einstein.ai/text/weather.csv'
    response = dataset.create_intent_dataset(path)
    print(json.dumps(response, indent=4, sort_keys=True))

if __name__ == "__main__":
    main()

Where listing for create_intent_dataset(..) is given below.

def create_intent_dataset(self, path):
  type = 'text-intent'
  return self._create_dataset(path, type)

This function in-turn calls _create_dataset(..)

def _create_dataset(self, path, type):
    multipart_data = MultipartEncoder(
        fields={
            'path': path,
            'type': type
        }
    )
    headers = {'Authorization': 'Bearer ' + self.access_token,
               'Content-Type': multipart_data.content_type}
    res = requests.post(LANG_DATASETS_URL + '/upload',
                        headers=headers, data=multipart_data)
    json_response = json.loads(res.text)
    return json_response

The HTTP POST request is made to the URL https://api.einstein.ai/v2/language/datasets/upload

Where constant LANG_DATASETS_URL is defined as show below:

LANG_BASE_URL = EINSTEIN_BASE_URL + '/language'
LANG_DATASETS_URL = LANG_BASE_URL + '/datasets'

Code Walkthrough
1. First we instantiate MultipartEncoder class, add path and type  values. type is text-intent in this case.

2. Instantiate headers json with Authorization and Content-Type

3. Call requests.post(...) to make the create call

4. Parse the response json and load it into json_response variable

Train the DataSet

Once the weather dataset is created we can train it using the Dataset class. Listing below shows how the DataSet is trained.

import json
from einstein.language.dataset import DataSet
from einstein.constants import ACCESS_TOKEN

def main():
    access_token = ACCESS_TOKEN
    id = '1010879'
    name = 'Weather Intent Model'
    dataset = DataSet(access_token=access_token)
    response = dataset.train_dataset(id, model_name=name)
    if('available' in response):
        print(json.dumps(response, indent=4, sort_keys=True))
    else:
        print('Response status ok?: ' + str(response.ok))
        print(json.dumps(response.text, indent=4, sort_keys=True))

if __name__ == "__main__":
    main()

Code Walkthrough
1. First we get the ACCESS_TOKEN
2. Assign a name to the model
3. Instantiate DataSet class with access_token in the constructor
4. Call dataset.train_dataset(..)
5. Extract json from the response and print

Predict an Intent

Once the model is ready we use it to predict the intent by giving the document and the model id.

import json
from einstein.language.prediction import Prediction
from einstein.constants import ACCESS_TOKEN

def main():
    access_token = ACCESS_TOKEN
    document = 'what is the weather in los angeles'
    model_id = 'V3E7CDCNLYMQD7XFM3NBXMZICE'
    prediction = Prediction(access_token=access_token)
    response = prediction.predict_intent(document, model_id)

    try:
        probabilities = response['probabilities']
        for x in probabilities:
            print(str(x['label']) + ":" + str(x['probability']))
    except TypeError :
        print('response ok? ' + str(response.ok))
        print('response content: ' + str(response.content))
    return True

if __name__ == "__main__":
    main()

Code Walkthrough

1. Get the access_token which is defined in einstein.Constants file

2. Initialize document and model_id

3. Initialize Prediction class

4. Call prediction.predict_intent(..)

5. Parse the response and list the probabilities

Output
On executing the code above an output is obtained which shows the intent type and its probability. Intent type is custom intent which is given as part of the training dataset. Higher the probability value, closer the input phrase is to the intent.

current-weather:0.9848461
five-day-forecast:0.009139863
hourly-forecast:0.0060140914

Summary

This blog post provides you with the basics on how to use Einstein Vision and Language APIs from a Python stand-alone application. Hopefully, these samples provide a good starting point for developers interested in extending their python applications to work with Einstein Vision and Language APIs. You can find the code repo for the library and samples here.

You can also try a project on Einstein Vision on Trailhead to learn how to integrate Einstein Vision into your Salesforce Platform workflows using Apex and Visualforce. If you have any questions, feel free to reach out on our forums.

About the author

Rajdeep Dua is a Director of Developer Relations at Salesforce. He is passionate about helping developers learn about cloud computing, machine learning, and Salesforce platform. He has over 18 years of experience in software product development and developer relations.

Related Links

• Salesforce Einstein Home Page
• Trailhead Quick Start: Einstein Image Classification
• Datasets used in this post on Github