Einstein Vision enables you to bring the power of image recognition to your application. As an example, this blog post describes how to use Einstein Vision to add image-based search to the DreamHouse sample application.

In the DreamHouse sample application, prospective home buyers know the type of house they like when they see it; but they may not know how that type of house is called (Victorian, colonial, Greek revival, and so on). This can limit their ability to search for houses based on their preferred architectural style. Using image-based search, they can search for houses based on a picture of a house they like.

Watch this video to see how to configure and use image-based search in the DreamHouse sample application:

Watch this video to see image-based search in action in the Salesforce1 mobile app:

To configure and use image-based search in your own instance of the DreamHouse application, follow these steps.

Step 1: Install the DreamHouse sample application

Follow these instructions to install the DreamHouse sample application. If you have a previous version of DreamHouse, make sure you install the new version (1.8 or higher).

Step 2: Create an Einstein platform account

If you already have an Einstein Vision account, skip this step and go straight to step 3.

  1. Go to the Einstein Vision signup site.
  2. Click Sign Up Using Salesforce.
  3. Enter your username and password and click Log In, then click Allow. You can authenticate with any org that you are a registered user of. To keep it simple, use your credentials from your DreamHouse org.
  4. On the activation page, click Download Key. A file named einstein_platform.pem is saved on your local file system.

Step 3: Upload your key file

  1. In the DreamHouse application, click the Files tab (depending on your screen size, it may be under the More option).
  2. Click Upload File.
  3. Select the einstein_platform.pem file you downloaded in step 2, and click Open. The einstein_platform file should appear in the list of files.
  4. In Setup, type Custom Settings in the Quick Find Box and click the Custom Settings link.
  5. Click the Manage link next to DreamHouse, and click the first New button to create default settings for the org.
  6. For Einstein Vision Email, enter the email address of the Salesforce user you used when creating the Einstein Vision key in step 2. You can leave all the other fields empty.

Step 4: Create and train an Einstein Vision dataset

  1. In the DreamHouse application, click the Einstein Vision tab. The Einstein Vision tab contains a custom component (EinsteinDashboard) that helps you manage your Einstein Vision datasets.
  2. Keep the default URL to the houses.zip file, and click the Create Dataset button. A new tile should appear for the houses dataset. houses.zip contains sample house pictures used to train the model. The house pictures are organized in three directories that Einstein Vision uses as labels: Colonial, Contemporary, and Victorian. Feel free to download and uncompress houses.zip to take a look at the directory structure.
  3. Click the Refresh Datasets button until you see the labels in the house dataset (Colonial, Contemporary, and Victorian). Note that there are 15 sample pictures by label. This is enough for this sample application, but in real life, you should add more sample images to increase the model accuracy.
  4. Click the Train button.
  5. Click the Models tab.
  6. Click the Refresh Models button several times until the progress column indicates 100% (the training process can take a few minutes).
  7. Select the model ID and copy it to your clipboard using Command-C (Mac) or Ctrl-C (Windows).

Step 5: Use the model to perform image-based search

In the Lightning Experience

  1. In the DreamHouse app, click the Property Explorer tab.
  2. Click the gear icon (upper right corner), then click Edit Page to open App Builder.
  3. Select the Image-Based Search custom component, and add it to the right sidebar of the page. Paste the model ID in the component property panel.
  4. Click Save and click Back.
  5. Drag an image of a colonial, Victorian, or contemporary house in the drop area of the Image-Based Search component. The component submits the image to the Einstein Vision service, which returns house type predictions. The PropertyTileList component then searches for houses matching the predicted house type.

Using the bot

DreamHouse comes with a bot custom component (in the utility bar) that lets you ask questions formulated in natural language in an instant messaging–like interface. For example, you can ask: “3 bedrooms in Boston” or just “find house.” Read this post to learn more about the bot custom component. A new bot command has been added to DreamHouse to support image-based search. Before you can use that command, specify your own model ID in the command handler:

  1. In the developer console, open the HandlerImageBasedSearch Apex class.
  2. Provide the value of your model ID for the modelId string at the top of the file.
  3. Save the file.

To use image-based search in the bot:

  1. Type “search houses like this” in the bot input field.
  2. Drag an image of a colonial, Victorian, or contemporary house in the drop area of the bot component.

In Salesforce1

A Lightning page named House Explorer is available to provide image-based search in the Salesforce 1 app. Once again, all you have to do is configure the Image-Based Search component on that page with your own model ID:

  1. In Setup, type App Builder in the Quick Find box and click the Lightning App Builder link.
  2. Click Edit for the House Explorer page.
  3. Click the Image-Based Search component.
  4. Paste your model ID in the component property panel.

To add the House Explorer page to the Salesforce1 mobile navigation:

  1. In Setup, type Navigation in the Quick Find box and click the Salesforce1 Navigation link.
  2. Add House Explorer to the Selected box.
  3. Click Save.

To use image-based search in Salesforce1:

  1. Tap House Explorer in the menu.
  2. Tap Upload Files.
  3. Select a picture of a colonial, Victorian, or contemporary house in your image library.

Wrapping up

Applications are getting smarter. With Einstein Vision, you can utilize image recognition to build artificial intelligence–powered apps fast. This blog post describes a simple visual search example, but the possibilities are endless. We can’t wait to see how you’ll bring the power of image recognition to your applications.

Resources

Get the latest Salesforce Developer blog posts and podcast episodes via Slack or RSS.

Add to Slack Subscribe to RSS