Apache Kafka has emerged as a next generation event streaming system to connect our distributed systems through fault tolerant and scalable event-driven architectures. This enables new types of intelligent and engagement applications, especially those that are powered by the new Salesforce Einstein technologies which brings AI to everyone.
Check out a demo of using Kafka to stream property view events from the DreamHouse web app and then consume those events in another app that processes the data and sends aggregates through a web socket to a real-time dashboard.
With Kafka you can build a event streaming topology that connects many event producers with many event consumers. While still using familiar publish / subscribe semantics, Kafka supports modern needs through message durability and horizontal scaling. Best of all you won’t need to manage your Kafka servers since Heroku now provides Kafka as a managed service.
Most Kafka systems ingest data from many sources including user interactions (app & web), telemetry data, or data change events (i.e. change data capture). Once ingested any
system can subscribe to the events on a named topic. For example, you may have a mobile app where every time a user views an item, that interaction is sent to Kafka. Then you might have a consumer which stores the raw events in a big data store. Another consumer could do real-time stream processing on the data to calculate aggregates and store those into a database so that dashboards or other visualization tools could have instant analytics.
To learn more about the Heroku Kafka service and how it relates to Salesforce, check out the Introducing Apache Kafka on Heroku: Event-Driven Architecture for the Cloud Era blog post.