Event-based integrations build context-aware, predictive, intelligent applications that operate at scale. Events are notifications that communicate data changes and messages between decoupled systems or services. This type of integration is an essential part of how Salesforce Data Cloud unifies customer data to build unified profiles of their customers.

Data Cloud’s infrastructure has to handle the complex task of synchronizing all data sources in real time, regardless of their origin. With eventing, information is rapidly requested and transferred between systems, enabling businesses to boost productivity and engagement by understanding each of their customers’ unique needs in real time throughout their entire journey.

Salesforce Pub/Sub API is a gRPC-based API that’s purpose-built for event-driven integrations that need to work at scale and in near-real time. Salesforce uses Pub/Sub API to connect Data Cloud’s event-driven architecture with the Salesforce event bus. Some key features of Pub/Sub API are:

• Event-driven: Enables real-time communication by allowing systems to react to events as they occur. Supports platform events, Change Data Capture events, and real-time monitoring events.
• Scalable: Bidirectional streaming via HTTP/2 allows for higher throughput with two-way simultaneous communication between client requests and the server. This enables Pub/Sub API to handle high volumes of events for large-scale applications.
• Single interface: One API enables event publishing, subscription, and event schema retrieval.
• Publish acknowledgments: Acknowledgements of published events after the events are stored in the event bus. The returned result is the final result of the publishing.
• Pull-based subscriptions: Pub/Sub clients control how quickly they want to consume events to prevent overflows during spikes in event publishes.
• Developer experience: Pub/Sub API simplifies the process of integrating Salesforce with other systems and services in any of the gRPC-supported programming languages.

Pub/Sub API and eventing use cases

Pub/Sub API lets you publish and subscribe to platform events, change data capture events, or real-time event monitoring that are already on the Salesforce event bus. That means it’s a new method for working with the events that are already in your org, making it a great fit for use cases like:

• Data synchronization: Keep data synchronized across different systems in real time. For example, update customer information in an external system whenever it changes in Salesforce using Change Data Capture.
• Automation orchestration: Flow and Apex can publish events for your Pub/Sub API client to consume. They can also subscribe and react to events that you’ve published with Pub/Sub API. For example, Flow can publish events when opportunities are won, or Apex can update cases based on events coming from external systems.
• Messaging and integration with external systems: Use platform events to integrate Salesforce with external applications and services, and make them communicate. For example, send an event to an external order management system when a new order is created in Salesforce.
• Real-time Event Monitoring and Alerts: Set up monitoring and alerting mechanisms. For example, send an alert to a monitoring system when a critical error occurs in Salesforce, or when a user logs into Salesforce.
• Analytics and reporting: Stream data to analytics platforms for reporting and insights. For example, send sales data to a BI tool for dashboards.

How Data Cloud scales with Pub/Sub API

Data Cloud uses Pub/Sub API as part of its Salesforce CRM Connector that syncs data from Salesforce CRM in real time to keep unified customer profiles up to date and calculate insights. Data Cloud is built on a data lake architecture, which is different from the transactional database in the Salesforce CRM. Bridging this gap with streaming events allows Data Cloud to handle a large volume of data changes and seamlessly replicate the data it needs. With scalability and latency as a key driver, events are a natural way to transfer new and updated data from Salesforce’s core applications to Data Cloud and other new services.

In building their Pub/Sub API client, the CRM Connector team took advantage of the greater throughput Pub/Sub API offers with bidirectional streaming and the increased control over event delivery offered by subscription flow control.

Let’s explore the key features of Pub/Sub API in detail, and delve deeper into how Data Cloud leverages this API for success.

Higher throughput with bidirectional streaming

Bidirectional streaming allows the client and server to send and receive messages independently and simultaneously over the same connection using gRPC over HTTP/2. Examples of bidirectional streaming in Pub/Sub API include publish acknowledgments and the subscription fetch request/responses.

Bidirectional streaming reduces latency between systems and supports efficient resource utilization as long-lived connections mean fewer connection setups and tear-downs, which can be resource-intensive. It also helps achieve higher throughput as a continuous data flow allows the system to handle multiple in-flight messages at the same time.

The server can manage multiple streams concurrently, distributing the load more evenly. It is an ideal choice for applications requiring high-frequency and instant data exchange, like real-time collaboration tools or financial trading platforms, where real-time data feeds are crucial for making timely decisions.

Furthermore, clients can process events from multiple topics in separate subscription streams by making multiple Subscribe calls (see docs) or multiple ManagedSubscribe (beta) calls (see docs) for different managed subscription IDs. When making multiple calls in one client, they have the flexibility to control the event flow separately for each subscription.

Increase observability with publish acknowledgements

Pub/Sub API offers a real-time final publish result for all Publish requests. This enhances the reliability of the Pub/Sub system because it ensures that messages are successfully received and processed in Salesforce. When a message is published, an acknowledgment is sent back to the publisher confirming receipt.

Publish acknowledgments also play a crucial role in error handling by providing immediate feedback if a message fails to publish. This helps quickly troubleshoot issues and allows for retries of failed messages. Events can be retried automatically or manually at the time of failure, ensuring that no data is lost due to transient issues. Additionally, detailed error messages and logs can be generated that help in diagnosing the root cause of the failure.

Flow control with pull-based subscriptions

Pub/Sub API is a pull-based API, which means that the client requests events from the server when it’s ready to process them. This gives the client control over the rate of event processing. Clients drive the rate of responses and can request more events at their preferred speed.

This kind of flow control helps ensure that clients are getting the latest events that they can process while still making informed use of their subscription limits. This contrasts with a push-based paradigm, where during a spike in events, clients can get overwhelmed with more events than they can handle and, as a result, lose events if they haven’t tracked the Replay ID of their last processed event.

Subscription flow control also helps developers know if they’re at the tip of the event stream. If they send a fetch request and get a partial batch, they know that there are no more events to consume. Pull-based flow control delivers an elevated developer experience since clients can manage the event delivery rate of a subscription-based on their processing speed; new events are only delivered when the system can handle the load.

Learn more about different flows that clients can use to rate control subscriptions in our developer documentation.

Conclusion

In this post, we covered how Pub/Sub API accelerates Data Cloud by supporting real-time data sync from Salesforce apps and orgs. We also discussed how clients can take advantage of the pull-based subscription pattern for greater control over their systems, how to use publish acknowledgments for monitoring and error handling, and how bidirectional streaming allows for higher throughput and scalability.

You can take Trailhead trails to start learning how to design event-driven apps, as well as more about events and Pub/Sub API. You can also begin by exploring the API and utilizing the prebuilt code examples to kickstart your next integration project. Additionally, there’s a demo app available that provides an end-to-end sample of an LWC application built using Pub/Sub API.

Keep an eye out for the next post in this series, where we’ll dive into the scalability performance of Pub/Sub API with some real numbers. Meanwhile, check out our previous blog post to learn about the new Managed Subscriptions (beta) feature for Pub/Sub API.

About the Author

Neha Ahlawat is a Product Management Director in the Einstein 1 Platform organization at Salesforce, where she is focused on the product areas of Metadata API, Tooling API, source tracking, and change sets. Follow her on X and LinkedIn.