Salesforce uses a central Group object to manage visibility related to the Role Hierarchy, Territory Hierarchy, Public Groups and Queues.  When administrative changes occur in these areas a group membership lock is taken to ensure data integrity is maintained while complex sharing calculations are completed.  The following activities take out group membership locks for the duration of their transaction:

• Role creation
• Role deletion
• Moving a role in the hierarchy
• Adding a user to a territory
• Removing a user from a territory
• Moving a territory in the hierarchy
• Territory deletion
• Territory creation
• Provisioning an internal user with an existing role
• User role change
• Provisioning a non-HVPU portal user under an account
• Portal Account owner change
• User Role change of a user who owns one or more portal accounts

In most cases, the Salesforce platform will try to obtain a lock for 10 seconds before failing.  When one of the above operations holds a lock for more than 10 seconds or the load is heavy enough to cause a queue of operations that exceed 10 seconds, an exception may be thrown by another operation trying to obtain the lock.  Following is an architectural strategy to manage locks for large enterprise customers who produce an extremely heavy load of group membership operations, usually through a combination of automated processes and heavy partner portal activity.  This strategy centers around ensuring business critical processes are always able to obtain group membership locks by organizing the operations that compete and stabilizing load on the group object.

The basis for this architectural strategy is to identify your business critical operations which must not be impacted by group membership lock exceptions.  All other competing operations which take a significant amount of time to complete must be processed serially, tuned and batched to ensure locks are not held longer than 10 seconds.  This ensures that your business critical operations will never have to wait long enough to produce an exception.  Here are the steps to setup this type of architecture:

1      Identify your business critical operations which must not be impacted by group membership lock exceptions.

• If operations are processed asynchronously and transaction time and load cause group membership lock exceptions by themselves, these operations will need to be tuned or processed synchronously relative to each other.  If this is an end user driven synchronous operation, individual requests may need to be batched and performed in ‘near real time’ so they don’t lock each other.

2      Evaluate which operations will compete for locks with your business critical operations. If you have the Granular Locking feature enabled, some group membership operations may proceed at the same time as your critical operations (contact Support to enable this feature). Possibilities are:

• Operations that only hold locks for a trivial amount of time (will not compete)
• All other Operations when Granular Locking is not enabled (will compete)
• Operations that can proceed in parallel under Granular Locking (will not compete)
• Operations that cannot proceed in parallel under Granular Locking (will compete)

3      Once you have identified which operations compete with your business critical operations, you can arrange the competing operations to be performed serially under the following conditions:

• Tune code and batch size of these operations to ensure group membership locks are never held longer than 10 seconds.
• Retry logic can be implemented for competing operations in case the business critical operation locks one of the competing operations.

4      Trim code related to business critical and competing operations which process group membership locks so that they perform only what is required for the group membership operation.  Additional code executed within the transaction holds the lock when the lock is not needed increasing the probability of exceptions.

Many organizations have user driven group membership changes which can compete with your business critical operations.  Since these changes cannot be throttled, business process should be implemented to perform significant changes during maintenance windows.  This will minimize their impact to your business critical operations.