As part of the Dreamforce ‘24 developer keynote, we demonstrated how to use the Models API (beta) to augment the Coral Cloud Resorts sample app (available on GitHub) securely using generative AI on the platform. This blog post takes a closer look at two specific use cases that were demoed during the keynote: creating SVG images dynamically and submitting feedback.

Models API use cases

The Models API allows you to invoke standard models offered by Salesforce or your own models connected via Model Builder, through the Einstein Trust Layer. It supports multiple use cases, and it has been exposed so you can build custom generative AI apps and workflows with code. If you haven’t heard about the Models API before, make sure to read our blog post on it (and Model Builder) before continuing.

As a summary, here are the capabilities that the Models API exposes:

• Generate text: You can send a prompt to the model and have it generate text. Using the Models API directly is ideal for text generation when you want to craft prompts dynamically in code and you don’t need the declarative customization capabilities of Prompt Builder prompt templates.
• Submit feedback: Provided you have audit and feedback activated (requires Data Cloud), you can use the Models API to submit feedback for a given generation (the different responses that the model produces). You can use this feedback later to further fine-tune your models and improve the quality of your prompts.
• Create a custom chat session: Using the Models API, you can create a custom chat session with the model, in which the history of messages is incorporated into each request, so the model has the full context of the conversation. When using this capability, on top of user messages, you can also send system messages, which are messages that your app crafts to provide additional context to the model.
• Generate embeddings: Embeddings are mathematical representations of content, such as words, pictures, audio, or video. Embeddings are stored in a vector database, where vectors represent the similarity of the content. The Models API can be used to generate embeddings for given text chunks. In this case, we recommend using an embeddings-specific model for the generation.

In this post, we’re focused on the first two capabilities.

The Coral Cloud Resorts demo

Coral Cloud Resorts is a sample hospitality application for a fictional resort that uses Agentforce, Data Cloud, and the Salesforce Platform to deliver highly personalized guest experiences. When guests arrive at Coral Cloud Resorts, they can register for various experiences available near the resorts.

For the demo, we created a Lightning Web Component (LWC) from which Coral Cloud employees can generate experience tickets using SVGs. We used the Models API to power two features:

• Dynamically generating SVGs using prompts when a user clicks the Generate button, while safely incorporating CRM data.
• Enabling users to give feedback for each generation of a ticket, good or bad, by clicking the thumbs-up or thumbs-down button.

Creating SVGs dynamically

The generation of the SVG makes use of the generate text capability we mentioned earlier. This is an ideal use case for the Models API, because the creation of the prompt is complex and done at runtime. The prompt changes dynamically depending on user selections (whether description, picture, and QR code are included or not), and the QR code is generated using a third-party JavaScript library. Here is a simplified version of the component’s JavaScript code:

Note: SVG generation via prompt templates API is not allowed, as multimodal generation is not yet supported.

The generateResponse method is called when a user clicks the “Generate” button on the LWC. Note that the prompt requests the response to just contain the SVG code, so we can easily extract it by removing the delimiters. Another approach when working with LLMs from code is to ask for JSON, and then parse the response using the JSON.parse JavaScript function.

The generateResponse method calls an Apex method named generateText , which we placed on a generic LLMService class that contains other methods to help us work with LLMs via the Models API and the Connect API.

As you can see from the code, you can specify the name of the model to use on the generation request (in this case, an Amazon Bedrock Claude 3 Sonnet model we connected through Model Builder). You can specify one of the models that are offered as a service by Salesforce, or a model that you’ve connected through Model Builder, as we did in this particular use case.

Note that we store the invocationId and generationId, so we can provide feedback later. At the moment it’s not possible to configure the Models API to ask for multiple LLM generations in a single invocation, but it is designed to support this capability in the future.

Providing feedback for the generation

The thumbs-up and thumbs-down feedback buttons on the LWC allow Coral Cloud employees to provide feedback regarding how the ticket generation went. These buttons are handled in JavaScript as follows:

And this is the implementation for the submitFeedback Apex method that is called when the thumbs-up and thumbs-down buttons are clicked:

The feedback data is then visible through out-of-the-box reports and dashboards, which query the GenAIFeedback__dlm DMO (Data Model Object) under the hood. You can query this, and other DMOs, in Apex using SQL or SOQL.

Note that to be able to log feedback, you need to have audit and feedback activated, and this requires Data Cloud and consumes Data Cloud credits.

Conclusion

As the two examples highlighted in this post illustrate, the Models API provides a convenient way of using LLMs through the Einstein Trust Layer in your code-crafted apps. It supports multiple use cases, such as text and chat generation, feedback submission, and embeddings creation, and it’s in beta as of October 2024.

Watch the Dreamforce ‘24 developer keynote to see the SVG and feedback use cases in action. To find more code examples and resources about prompt templates and the Models API, check this GitHub repo, where you’ll find the developer guides and references, the existing Postman collections for both APIs, additional videos, and some Trailhead modules to explore.

About the authors

Alba Rivas works as a Principal Developer Advocate at Salesforce. You can follow her on GitHub or LinkedIn.

Charles Watkins is a Lead Developer Advocate at Salesforce. You can follow him on LinkedIn.