Visual Search using SPI

With the Point of Sale Visual Search SPI, the POS app sends an image captured by a store associate to an external image-recognition provider and receives a ranked list of matching product IDs. Results are resolved against the local product catalog and displayed in the POS app. Optionally, the provider can return filters, sort options, and pagination to enable server-side refinement of results.

  • Modern POS app build: 10.7.1 or later
  • Platform support: iPad and iPhone
  • Offline support: No

For a complete list of Point of Sale APIs and SPIs, see Point of Sale API, SPI, and Webhook Overview.

This diagram shows how the store associate, Point of Sale, and the SPI implementer interact during a visual search session.

Sequence diagram showing the interaction flow between the store associate, Point of Sale, and the SPI implementer during a visual search session.

  1. The store associate captures an image in the POS app.
  2. Point of Sale sends the image to the SPI provider via the VISUAL_SEARCH_UPLOAD connector.
  3. The provider performs image recognition and returns a searchId along with a ranked list of matching product IDs. Optionally, the response can include filters, sortOptions, and pagination. See POS Experience Based on SPI Response.
  4. Point of Sale resolves the product IDs against the local product catalog and displays results to the store associate.

If the SPI response from step 3 includes filters, sortOptions, or pagination, and the VISUAL_SEARCH connector is configured, Point of Sale renders the corresponding UI controls (filter panel, sort dropdown, page navigation).

  1. The store associate applies a filter, changes sort order, or navigates to the next page.
  2. Point of Sale sends a request to the SPI provider via the VISUAL_SEARCH connector, passing the searchId and the selected criteria.
  3. The provider applies the criteria to the stored result set and returns an updated list of product IDs (with optionally updated filters, sort, and pagination reflecting the new state).
  4. Point of Sale displays the refined results.

The second SPI call is entirely optional. If your provider doesn’t support server-side filtering, sorting, or pagination, omit filters, sortOptions, and pagination from the upload response. The POS doesn’t render these controls, and the VISUAL_SEARCH connector doesn’t need to be configured.

Configure one or both SPI connectors in CMS depending on whether your provider supports server-side refinement.

  1. In CMS, go to Integrations | Custom Connectors.

  2. Click Create.

  3. For Connector Type, select SPI.

  4. For Endpoint Name, select VISUAL_SEARCH_UPLOAD.

  5. For API Endpoint, specify the external endpoint for image upload and recognition.

  6. For Integration Type, select MULTIPART_FORM_POST.

  7. Specify any required JSON headers. For examples, see Point of Sale API, SPI, and Webhook Overview.

  8. Configure the remaining fields as needed.

  9. Save your changes.

Configure this connector only if your SPI provider supports server-side filtering, sorting, or pagination.

  1. In CMS, go to Integrations | Custom Connectors.

  2. Click Create.

  3. For Connector Type, select SPI.

  4. For Endpoint Name, select VISUAL_SEARCH.

  5. For API Endpoint, specify the external endpoint for search refinement.

  6. For Integration Type, select HTTP_POST.

  7. Specify any required JSON headers. For example, {"Content-Type":"application/json"}.

  8. Configure the remaining fields as needed.

  9. Save your changes.

Enable the populateNewCustomAPIHeaders toggle to inject contextual headers into every SPI request. For details, see Headers Sent by Point of Sale.

The POS experience varies depending on the content of the SPI response.

With a minimal response, the store associate sees a flat list of visually similar products. No pagination, filters, or sorting controls appear. This is the simplest implementation: image in, products out.

With a full response, the store associate sees product results with page navigation, a filter panel, and a sort dropdown. Selecting a filter, changing sort, or navigating pages triggers a call to the VISUAL_SEARCH connector.

Each entry in filters[] becomes a section in the filter panel. The displayName value is the section title. Each entry in values[] renders as a selectable option. The order of values in the array determines the display order.

The filter panel in the POS app showing Category and Size sections with selectable options.

The sortOptions.options[] array renders as a dropdown list. Each option’s label is the row text. The key is the value sent back to the SPI when selected. The option whose key matches sortOptions.current is pre-selected.

The sort dropdown in the POS app showing Relevance selected, with Price: Low to High and Price: High to Low as additional options.