Getting Started with the Force.com Streaming API

Abstract

The Force.com Streaming API lets you expose a near real-time stream of data from the Force.com platform in a secure and scalable way. Administrators can create topics, to which applications can subscribe, receiving asynchronous notifications of changes to data in Force.com, via the Bayeux protocol.

This article explains the basics of the Streaming API and shows how to get started using it from Java client applications and Visualforce pages.

NOTE - you can also see a simple and quick demo by installing the Unmanaged Streaming API Demo package.

NOTE - this article has been updated for the Winter '12 release (API version 23.0). The Streaming API documentation contains a code sample that can be configured to work with both API version 23.0 and the previous release, 22.0).

Streaming API Overview

The Force.com Streaming API uses a publish/subscribe model where administrators create one or more named topics, each of which is associated with a SOQL query. Applications may subscribe to one or more topics, using the Bayeux protocol. As relevant data is updated, the platform re-evaluates the query and, when a change occurs that alters the results of the query, the platform publishes a notification to subscribed applications.

The Streaming API is secure; client applications must provide a session ID (sid) in all interactions. This article will show you how to secure both JavaScript in a Visualforce page and Java application communication with the API.

If your application is currently polling the SOAP or REST API to detect changes, you should consider refactoring it to use the Streaming API; you will see better performance with lower resource consumption.

For technical documentation, and details on how to join the Streaming API Developer Preview, check out the Streaming API Resource Page.

Setting up for the Sample Application

For both samples, you'll need

For the Java application sample, you'll also need

Using the Force.com Streaming API

This section walks you through a simple scenario of using the Streaming API. We begin with the basics - establishing a query and creating topics, and then show two client scenarios - one based on Java and another on Visualforce.

Query

The first step in using the Streaming API is to define a SOQL query that will return the records you are interested in. For example, your application may require a notification whenever a new Account is created. A suitable query might be:

SELECT Id, Name FROM Account

Alternatively, you might only be interested in Accounts with more than 1,000 employees:

SELECT Id, Name FROM Account WHERE NumberOfEmployees > 1000

As a special case, you can specify a record id in the WHERE clause and receive notifications whenever any change occurs on that record:

SELECT Id, Name FROM Account WHERE Id = '001x0000002JSofAAG'

These are just some simple examples - see the Streaming API documentation for a full discussion of Streaming API queries.

PushTopic

Once you have decided on your query, it's time to create a PushTopic that binds a topic name to the query. You can set the following fields on the PushTopic record:

Field Name Type Example Description
ApiVersion double 23.0 Required. API version to use for executing the query specified in Query.
Description string All records for the Account object. Optional. Description of what kinds of records are returned by the query.
Name string NewAccounts Required. Descriptive name of the PushTopic. The maximum length is 25 characters.
Query String SELECT Id, Name FROM Account Required. The SOQL query statement that determines which record changes trigger events to be sent to the channel. Maximum length is 400 characters.

There are also several system fields on PushTopic such as CreatedById and SystemModstamp.

As soon as you create a PushTopic record, applications can subscribe to the topic.

Go ahead and create a topic now. Login to your Developer Edition environment, navigate to the System Log (Your Name > System Log), and paste the following code into the 'Execute Apex' area:

PushTopic pushTopic = new PushTopic();
pushTopic.ApiVersion = 23.0;
pushTopic.Name = 'AllAccounts';
pushTopic.Description = 'All records for the Account object';
pushtopic.Query = 'SELECT Id, Name FROM Account';
insert pushTopic;
System.debug('Created new PushTopic: '+ pushTopic.Id);

You should see the id of the new PushTopic record in the log output. You don't need to keep a note of that id - you'll use the PushTopic Name - but it's a useful confirmation that you successfully created the record.

Publish/Subscribe with the Streaming API

Now that you've created a topic, your applications can subscribe for updates. Currently, apps use the Bayeux protocol to make a connection to the Streaming API, subscribing to topics via the long-polling transport over HTTP. It's unlikely you'll need to code to the Bayeux protocol directly, though - the CometD project provides open source Java and JavaScript implementations; there are also other open source implementations available for a range of other languages including Ruby, PHP and Perl.

Receiving Notifications in a Visualforce Page

Now you can create an application that will receive notifications for your newly created topic. The simplest example is a Visualforce page, since the user will already have a session, and it is straightforward to pass the session ID as the Authorization HTTP header.

The first step is to extract the CometD JavaScript library file. Download CometD, unpack the tarball, then:

$ cd cometd-2.2.0/cometd-javascript/common/target
$ jar xvf cometd-javascript-common-2.2.0.war org/cometd.js

Next, upload the following files as static resources (Your Name > Setup > Develop > Static Resources > New):

File Resource Name
cometd-2.2.0/cometd-javascript/common/target/org/cometd.js cometd
cometd-2.2.0/cometd-javascript/jquery/src/main/webapp/jquery/jquery-1.5.1.js jquery
cometd-2.2.0/cometd-javascript/jquery/src/main/webapp/jquery/json2.js json2
cometd-2.2.0/cometd-javascript/jquery/src/main/webapp/jquery/jquery.cometd.js jquery_cometd

Now you have your resources in place, you can create a Visualforce page that will receive notifications. If you have not yet enabled 'Development Mode' for your user then do so now - Your Name > Setup > Administration Setup > Manage Users > Users > Your Username > User Detail - it makes working with Visualforce pages much easier.

With development mode set, you can create a Visual Force page by modifying the URL in your browser to https://instance/apex/StreamingPage, where instance is the Force.com instance you usually see in the console, for example, na1.salesforce.com, and clicking 'Create Page'.

SFDC-NewStreamingPage.jpg

In the resulting new page, click the icon in the bottom right of the window to open the edit pane and paste in the following code.

<apex:page>
    
<apex:includeScript value="{!$Resource.cometd}"/>
    
<apex:includeScript value="{!$Resource.jquery}"/>
    
<apex:includeScript value="{!$Resource.json2}"/>
    
<apex:includeScript value="{!$Resource.jquery_cometd}"/>
    
<script type="text/javascript">
    
(function($){
    
    $(document).ready(function() {
    
        // Connect to the CometD endpoint
    
        $.cometd.init({
                url: window.location.protocol+'//'+window.location.hostname+'/cometd/23.0/',
                requestHeaders: { Authorization: 'OAuth {!$Api.Session_ID}'}
            });

            // Subscribe to a topic. JSON-encoded update will be returned 
            // in the callback
            $.cometd.subscribe('/topic/AllAccounts', function(message) {
                $('#content').append('<p>Notification: ' +
    
                'Channel: ' + JSON.stringify(message.channel) + '<br>' +
    
                'Record name: ' + JSON.stringify(message.data.sobject.Name) + '<br>' + 
    
                'ID: ' + JSON.stringify(message.data.sobject.Id) + '<br>' +
    
                'Event type: ' + JSON.stringify(message.data.event.type)+'<br>' +
                    'Created: ' + JSON.stringify(message.data.event.createdDate) + '</p>');
    
        });
    
    });
    
})(jQuery)
    
</script>
    
<body>
    
<div id="content">
    
<p>Notifications should appear here...</p>
    
</div>
    
</body>
</apex:page>

Hit save and verify that the page saved without errors. Now open another browser window and create a new Account. You should see a notification appear in the Visualforce page:

SFDC-Notification.png

If you don't see a notification within a few seconds, enable your browser's debugging console (in Firefox 4, Web Console, which is on either the Tools or Web Developer menu, depending on platform; in Chrome, View > Developer > JavaScript Console; in Internet Explorer 9, F12 Developer Tools) and reload the page. Often the error is apparent from the debug output - a missing script, for example.

A real application would do much more than simply display the new Account name and associated data. For example, if the Visualforce page was showing a list of Accounts, the new Account could be dynamically added to the list, immediately available to the user.

Receiving Notifications in a Java Application

On-premise applications written in Java and other languages can also receive notifications via the Force.com Streaming API. In this section you'll create a Java application to receive updates from that same topic you created earlier.

  • Create a command-line Java application project using your IDE of choice.
  • Copy the following JARs from the CometD 2.2.0 implementation to your project:
    • cometd-2.2.0/cometd-java/bayeux-api/target/bayeux-api-2.2.0.jar
    • cometd-2.2.0/cometd-java/cometd-java-client/target/cometd-java-client-2.2.0.jar
    • cometd-2.2.0/cometd-java/cometd-java-common/target/cometd-java-common-2.2.0.jar
  • Log in to Force.com environment, navigate to Setup ➤ Develop ➤ Remote Access, and click New to create a new remote access application .

StreamingRemoteAccess.png

  • Since your application will be using OAuth 2.0 username/password flow to login, just enter a dummy value for Callback URL.
  • After clicking Save, you will see your new application’s credentials:

StreamingRemoteAccess2.png

  • Click the link to reveal the consumer secret. Note – OAuth 1.0 terminology is currently used in the Remote Access screen. The OAuth 2.0 specification uses ‘client’ in place of ‘consumer’.
  • Copy the consumer key and consumer secret, and set them as environment variables in the project settings in your IDE. For example, in Eclipse Helios, right click the project, and select Properties > Run/Debug Settings > select the project > Edit > Environment. You will also need to add your username and password, plus the Force.com login URL for your org; by default this will be https://login.salesforce.com/ for Developer Edition orgs, https://test.salesforce.com/ for sandboxes.

EnvironmentVariables.png

Note that it is best practice to keep credentials such as OAuth client identifiers and secrets and API usernames and passwords out of source code. Using environment variables achieves this end with a minimum of extra work. You'll also need to set the topic name as a command-line parameter.

CommandLineParams.png

  • Now run the sample. If everything is working, the application will authenticate via OAuth 2.0 username/password, do the Bayeux handshake, subscribe to the topic, and wait for updates:
Running example....
Login response: {
  "access_token": "00Dx00000008t03!AR0AQGjuKpdpiqFYnNHDJ2.GIjcSmBb3Zp0MbfNJR1hnMD9FhwU.8sF3ty4fuQ2R.13c3PN_Lno57qiDfvjzSaOYpFZVWvHM",
  "id": "https://prerellogin.pre.salesforce.com/id/00Dx00000008t03EAA/005x0000000JNgnAAG",
  "instance_url": "https://prerelna1.pre.salesforce.com",
  "issued_at": "1308678916406",
  "signature": "dpQvS25GyXgB3fi2KlvTKHT/cg1KM7AgDn2apNvgkjw="
}
Waiting for handshake
Subscribing to topic: NewAccounts
Waiting for streamed data from salesforce...
  • Now go back to the browser and create another Account; you should see it reflected in the application output:
...
Waiting for streamed data from salesforce...
Received Message: {
  "channel": "/topic/NewAccounts",
  "data": {
    "Id": "001x0000002JcsJAAS",
    "Name": "Another New Account",
    "eventCreatedDate": "Tue Jun 21 17:55:32 GMT 2011",
    "eventType": "created"
  }
}

Again, a real application would do much more than simply display the notification message; for example, if the query was on some field value exceeding a critical threshold, the application might send an SNMP alert or SMS message.

Bonus Java Sample

MakeTopic.java shows you how to create a PushTopic record from Java using the Force.com REST API. You'll need the same Jetty libraries as the Java client example app above.

Summary

This tutorial showed how to use the Force.com Streaming API. The source code demonstrates how to use the API from JavaScript running in a Visualforce page and a Java command-line application, though the techniques can be transferred to other programming languages.

The Force.com Streaming API is a powerful, convenient, and simple way to obtain a near real-time stream of data from the Force.com platform in a secure and scalable way. We look forward to seeing our developer community use the REST API to create new innovations around the Force.com platform.

References

About the Author

Pat Patterson is a member of the Developer Evangelism team at Salesforce.com, currently focusing on dynamic languages. Describing himself as an 'articulate techie', Pat hacks all manner of code from Node.js web apps down to Linux kernel drivers, writing it all up on the Force.com blog, his own blog Superpatterns, and, of course, Twitter.