Force.com Asynchronous Processing Imagine an assembly line in a manufacturing plant. On a normal day, parts are continuously fed onto the line while hundreds of workers do their job, efficiently putting together those parts to build the final product.

Now imagine that a team of mechanics arrives one day, unannounced, during the middle of peak work hours. They start shutting down various sections of the line to perform maintenance on the machinery. Due to a lack of communication, parts keep getting fed onto the line, but the workers can’t do their jobs because of the interruption caused by the mechanics. To make matters worse, some of the mechanics make a bunch of mistakes, extending the maintenance work. The line becomes a mess, workers get frustrated, and production grinds to a halt–all because of poor planning around when the (inefficient) maintenance operation happens.

Good planning and coordination among manufacturing plant teams prevents mishaps like this one. Properly trained mechanics communicate with assembly line managers and schedule maintenance operations so that they happen when the plant is shut down and don’t interfere with normal work hours, thus preserving the productivity of the assembly line.

Assembly Line? Salesforce?

The assembly line story is a useful analogy for a problem we often see when helping customers troubleshoot problems that originate from the use of the platform’s asynchronous processing tier–problems that cause transaction performance and page response time issues on Force.com during peak processing hours. Imagine …

the assembly line = Force.com transaction processing engine
the assembly line workers = Salesforce users executing peak-time daily transactions
the unannounced assembly line maintenance operations = Force.com bulk jobs that use the platform’s asynchronous processing mechanisms
the assembly line mechanics = Force.com admins, developers, or architects that scheduled bulk jobs in the middle of the work day without regard to transaction load
the mechanics mistakes = unnecessary inefficiencies in the bulk jobs

This post is a quick survival guide that can help you to avoid thread starvation and resource load competition, maintain transaction response times, and preserve the productivity and happiness of your org’s users.

Step 1: Learn About Force.com Asynchronous Processing

If you plan to use Force.com to support your business, it’s in your best interest to learn all that you can about how the platform functions and how to best architect your implementation and applications. The resources on the Architect Core Resources page explain key concepts and best practices to be aware of about Force.com. In the context of this post, don’t miss Asynchronous Processing in Force.com, which explains the underlying concepts of asynchronous processing and best practices that you can follow to build effective design patterns around asynchronous processing. I highly recommend that you read this short paper front to back–it’s a very interesting under-the-hood look at Force.com.

Thread Starvation and Resource Load Competition

As you read this paper and explore Force.com’s asynchronous processing architecture, you come to learn that asynchronous requests, especially those that are inefficient, can create or exacerbate two detrimental conditions of which you should be aware.

Thread Starvation: Large requests can consume all or a portion of the asynchronous processing threads available to your org.
Resource Load Competition: Large requests can place a tremendous amount of load on the platform, which can detract from the performance and response time of other asynchronous processing jobs executing for your org.

Asynchronous Request Types

Carefully consider what types of requests Force.com’s asynchronous processing layer handles to start your journey on avoiding thread starvation and resource load competition. On Page 1, the paper states “There are over 200 different types of asynchronous requests on Force.com …,” and then goes on to use Apex Batch, @future requests, and Bulk API jobs as examples of common asynchronous requests. Here’s a short list of some other, perhaps less obvious, request types that compete for the asynchronous processing resources available to your org.

Dashboard refreshes
Sandbox copies
Weekly exports
Sharing rule recalculations
Organization-wide sharing default changes
Web to X (e.g., Web to Case, Web to Lead)
Email to X (e.g., Email to Case, Email to Lead)
Roll up summary field recalculations or modifications

Don’t Forget Custom Object and Field Deletions

Deleting custom objects and fields leverages asynchronous processing, but in a delayed fashion. For example, when you delete a custom object, Salesforce “soft deletes” the object and corresponding data. Approximately 15 days later, a background process enqueues the object and its data for hard deletion using the asynchronous processing layer. Just like those untimely assembly line maintenance operations that interfere with production, the database locks held during hard deletions have the potential to create lock contention that could interfere with ongoing transaction processing.

Step 2: Design Force.com Asynchronous Jobs With Care

The document referenced above covers a lot of great information about design practices for the Force.com asynchronous processing tier. Here are a few other design tips to keep in mind so that you match up your requirements appropriately.

Time-Sensitive Business Functions

Only use Force.com asynchronous processing features for business functions that are not time sensitive. Although the vast majority of Force.com asynchronous jobs execute in a timely fashion, the platform dequeues jobs for processing based on current load, which can vary tremendously given that asynchronous work often corresponds to long-running bulk jobs. Consequently, there’s no guarantee or SLA as to when queued jobs will execute or finish.

Refactor When Necessary

If you have time-sensitive processes that you can’t architect on the synchronous framework due to platform governor limits, consider breaking these processes up as follows:

processing time-sensitive components synchronously
processing time-insensitive components asynchronously

Another alternative is to build an on-premise or Heroku application to execute heavy processing, and then integrate with Salesforce via our APIs.

Step 3: Test, Tune, Test, Tune …

Long-running asynchronous payloads can contribute to thread starvation and resource load competition; therefore, it’s critical to tune the efficiency of large bulk jobs so that they execute as fast as possible and free up more asynchronous processing resources for your org. This best practice applies to all types of bulk jobs, especially recurring jobs.

For example, which one of the following designs do you think would be more efficient for refreshing an object that contains millions of records.

Deleting all records and then inserting all current records.
Updating only those records that have changed since the previous refresh, then inserting new records.

Hopefully you chose #2. Processing deltas, especially when working with large volumes of data where a small percentage records require updates, is the most efficient approach.

Test in Sandbox

Testing and tuning of all new Salesforce business processes, including bulk processes, should happen in sandbox. Although you’ll likely see substantial variation in runtimes due to variable server loads and differences between sandbox and production pods, it’s a good idea to have a general idea of how long it’ll take to complete a job so that you can schedule jobs with awareness.

Step 4: Schedule Force.com Asynchronous Jobs with Awareness

Awareness inevitably leads to better decisions. For example, once you understand thread starvation and resource load competition, you are more likely to carefully consider the volume and timing of your asynchronous org’s requests.

Here are some basic considerations for the execution of Force.com asynchronous requests.

Be Aware of the Criticality of Asynchronous Requests

During your org’s peak transaction processing hours, limit time-insensitive asynchronous requests as much as possible so that there’s more capacity to handle other requests. This rule might translate into things like moving a bulk data load to off-peak hours, scheduling a massive sharing rule recalculation to the weekend, or increasing the refresh interval for dashboards.

Be Aware of Completion Times for Asynchronous Requests

Once you tune jobs, understand their time-sensitivity, and have a general idea of how long they’ll take to complete, figure out where they can run best. For example, to avoid conflicts with other requests, you might schedule time-insensitive asynchronous requests as follows.

for jobs that take 0-12 hours to complete, you might start them any evening after 8pm
for jobs that take 12+ hours to complete, start them Friday evening after 8pm

Be Aware of Other Asynchronous Requests

Again, your org has access to a limited amount of asynchronous processing resource at any given time. Don’t unnecessarily create thread starvation or resource load competition by scheduling too many things to happen all at one time. Build and share a calendar that provides your team with visibility of asynchronous requests throughout the day and make it policy to use that calendar for scheduling new events.

Be Aware of Things Out of Your Control

What about those custom object and field deletions–you don’t have control over when the final deletion step happens? In the case of large objects or field deletions for objects that have millions of records, open a case with salesforce.com Technical Support to schedule this step during off-peak hours. Follow this process whenever you are unsure of something that might have an impact on both synchronous and asynchronous processing–better safe than sorry.

Summary

Force.com’s asynchronous processing tier is a powerful platform resource that executes many types of processes in the background. Before leveraging any one of a number of asynchronous processing features such as the Bulk API, Apex batch jobs, and @future requests, take the time to learn about the platform’s asynchronous processing architecture and how to best use related features in concert with one another and avoid unnecessary pitfalls. Your users will be happy you did.

Related Resources

About the Author

Steve Bobrowski is an Architect Evangelist within the Technical Enablement team of the salesforce.com Customer-Centric Engineering group. The team’s mission is to help customers understand how to implement technically sound Salesforce solutions. Check out all of the resources that this team maintains on the Architect Core Resources page of Developer Force.