Salesforce1 Platform Enterprise Environment Management

Editors Note: This post is part of a “Guest” series entitled Enterprise Architecture with Force.com. Our guest blogger, Greg Cook, is a managing partner of CloudPremise and currently holds all seven Salesforce certifications.

Establishing an effective Environment Management strategy is critical for utilizing the Salesforce1 Platform.  Salesforce.com has already established itself as a leader in cloud technology and innovation, however some aspects of dealing with the platform still require good-old-fashioned IT management skills.  Salesforce provides a LOT of content around governance, environment management, and change management.  However I have noticed at multiple clients the information and options are sometimes too plentiful.  Many customers want to be given a recipe and a framework as opposed to designing their own solutions from best practices.  This installment describes a typical environment management strategy that can be emulated on many Enterprise projects.

What do I mean by Enterprise Projects?  This article applies to your organization if you meet the following criteria:

1. Your Salesforce landscape is made up of one or more orgs consisting of “high complexity” (lots of configuration, lots of code, lots of data).
2. Your Salesforce projects are required to follow a formal change process under the governance of your Enterprise Architecture and/or ITSM frameworks.
3. Your Salesforce projects consist of many resources including architects, business analysts, configurators, developers, and testers.
4. You have have a large number of requirements often prioritized by competing business stakeholders.
5. You have active system administrators who are making authorized (and sometimes unauthorized) changes directly in production.
6. You constantly have issues deploying changes to production.

If this sounds familiar than you are in luck and this article was written for you.  If your projects do not sound like the above list then many of the recommendations in this article may not apply to your environment (but feel free to keep reading!)

What You Should Know About Environment Management

The first thing to understand about environment management on the Salesforce1 Platform are some of the differences to typical environment management.

1. Your Salesforce Production Org is Your Only “Pristine” Environment

In traditional software development your pristine environment can be housed within source control and configuration management tools. In order to maintain a stable production deployment the emphasis is using configuration management techniques to maintain your code, configuration files, and deployment scripts.  Production can be rebuilt, deployed at will, and rolling back to a previous version of the code is possible.  However in Salesforce it is not possible to take production offline or to deploy the entire application in a big-bang deployment event.  Instead you are trying to migrate “differences” between your environments into production.  You can utilize source control and configuration management tools with Salesforce, however your production environment is (almost) always live regardless of the state of your configuration management procedures.

2. Salesforce is a heavily configured environment – much of which can be done directly in production

Salesforce supports a much higher volume of “production changes” due to its metadata based configuration design.  Therefore many changes can be done safely in production without the need for a code deployment.  While this has the effect of producing immediate value to the business (Admin Hero’s anyone?), it can have dreadful consequences on future deployments if the right controls are not in place.

3. The larger the difference between environments directly correlates to the difficulty in migrating between those environments

As I already described, it is not possible to move the entire code base from one state to another (i.e. Dev –> QA –> Prod, etc).  The nature of Salesforce’s migrations means that the emphasis should not be on your configuration management artifacts, but rather the emphasis should be on environment synchronization.  You may have the best code and most thorough migration plan – but if your environments are out of sync with production changes you will have a VERY difficult time deploying changes of significant complexity.

4. Your sub-prod environments differ from your Production environment

This is NOT uncommon on traditional projects.  However there are specific pain points around Salesforce sandboxes.  The most noticeable difference will be the database size.  Many customers do not even have a full copy sandbox, let alone multiple.  Most sandboxes will not come loaded with data, and those that do may need changes to support testing and integration.  There are other differences outside the scope of this article, however the main pain point remains: there will be overhead in each sub-prod environment to manage data and metadata.  This overhead increases as the number of sandboxes increase.

5. Not all changes to Salesforce can be automated via the API

Salesforce is releasing significant new features three times per year.  While many of these new features can be managed and maintained using the APIs, some changes are not possible except through the web-browser interface.  That means that maintaining a strict documentation set is VITAL for successful migration and environment maintenance.

With those prerequisites out of the way it is time to introduce my reference model for Enterprise Environment Management:

Let’s walk through the diagram and explain the concepts and purpose of each component, including the types of change to each environment.

#1 – Production

As mentioned, this is your only pristine environment.  Refreshes are only available from production into the sandboxes (reflected by the red lines).  Therefore your code and configuration changes are “swimming upstream” – trying to migrate successfully to the production environment from the lower sub-prod environments.

• API Based Deployments – Your production deployments should take place mostly through a strictly controlled process via ANT or Change sets.  The only environments I would allow to migrate into Production would be the System Test (#6), Stage (#7), or PFix (#9) environments.
• Manual Changes (Configuration and Data) – Any configuration changes made to production must be applied to sub-prod environments either manually or via a refresh.  Subsets of production data must also be pushed into multiple sandboxes to support testing, training, and integration.
• Refreshes – You can and should refresh sub-prod environments often.  If you do NOT refresh your sub-prod environments you have 2 choices: 1) manually maintain ALL changes from Production to each “managed” sub-prod environment or 2) risk very difficult and unpredictable deployments.

#2 – Un-managed Developer Environments

Each developer on the project will need an environment.  Typically these environments are Developer Sandboxes.  The care and feeding of this environment is usually done by the developer themselves.  These environments typically only exist for the length of time it takes to migrate a feature to production, and sometimes even less.  I call these “unmanaged” environments because typically they will not be maintained by your environment manager.  A developer should complete Unit testing and apex-based automated testing in this environment.

• Refreshes – I typically expect these sandboxes to be refreshed after a working feature has been successfully migrated upstream.
• Manual/API Maintenance – Changes from other developers (and eventually other projects) must be successfully applied to each developer sandbox.  This “merging” of configuration and code can be done multiple ways including source code tools, change sets/metadata API, or manually.  This step is very important depending on the number of developers and the number of parallel projects inside a single org.
• API Based Deployments – Working features should be deployed upstream using Change Sets or ANT.  Get in the habit of maintaining a strict manifest (list, index, catalog, etc) of components that need to be migrated.

#3 – Managed Project Environments

Each project needs to have an environment that can isolate changes from other projects.  Many projects may be occurring at the same time and on different release schedules.  These managed environments are the holding grounds for project features that are not ready for release.  This is where I would have testers complete Functional testing.  I call these “managed” environments because typically they are (or should) be maintained by an environment manager (a system administrator with responsibility for environment integrity).

• Refreshes – Typically these sandboxes would only be refreshed after a project has migrated upstream.  Sometimes it is necessary to refresh a sandbox prior to this upstream migration, in which case all project changes would have to be reapplied.  Source control tools are great for this; however expect manual maintenance as well.
• Manual Changes (Configuration and Data) – Any configuration changes made to production must be applied to sub-prod environments either manually or via a refresh.  Sub-sets of data must also be pushed into sandboxes to support testing and integration.
• Manual/API Maintenance – Changes from other projects must be successfully applied to each project sandbox.  This “merging” of configuration and code can be done multiple ways including source code tools, change sets/metadata API, or manually.  This step is very important depending on the number of parallel projects inside a single org.  This step becomes more or less important based upon the release calendar of each project, and whether the project sandbox can afford being refreshed mid-stream.
• API Based Deployments – When a project is ready to move into production it should be migrated upstream into the “release train”.  Changes should be migrated using Change Sets or ANT.  Get in the habit of maintaining a strict manifest (list, index, catalog, etc) of components that need to be migrated.

Note: I have included Citizen Developer Environments in this region.  Citizen Developers would be small projects that do not require multiple developers/sandboxes.  However it is important to have the Citizen Developers migrate into production using the same path as your enterprise projects (i.e. the release train).

#4 – Managed Release Train

I use the concept of a release train to describe the activities necessary to migrate successfully to production.  A release train can be a simple schedule detailing when changes can be made to each environment, or it can be a complex automated solution.  You can read about release trains more here.  For the purpose of this article I will use a very simple example of a release train:

• Only changes bound for production should be on the train.  Therefore make sure your unit and functional testing in the lower environments is very thorough.
• Only on a predefined basis is movement allowed from one “car” to the next.  For example, a weekly cycle (i.e. Tuesday nights, etc) would be change window in which migration could occur on the train.  The cadence would be set based upon your organization’s capacity to support deployment activities.  It is important to set a precedence on release train migrations of “often enough but not too often” as well as “highly disciplined and predictable”.
• The number of environments inside your release train will be determined by the types of gates necessary to support your deployment process.  I have recommended three stops (integration, test, stage) but this maybe more or less depending on your specific organization.

Let’s look deeper at each stop on the release train:

#5 – System Integration

This is the environment where parallel projects comes together.  This is also a good spot for system integration testing.  Automated apex tests should be continually run in this environment to ensure a project has not broken something.  Consider the use of more sophisticated automation tools to ensure your business processes are protected.  Selenium is a good open source tool for testing the user interface.  You can also test APIs and integrations with tools like SoapUI or Postman.

• Refreshes – I would refresh the system integration sandbox as often as is practical.  Therefore I would NOT recommend a full-sandbox with its accompanying 29 day refresh limit.
• Manual Changes (Configuration and Data) – If you will not refresh your sandbox then expect a much higher level of manual maintenance.  All production changes must be pushed into this environment one way or another.
• Manual/API Maintenance – If necessary code can be pushed from System Integration “back down” into a lower sub-prod environments.  This would allow parallel projects to integrate sooner.  This is a great technique if you know there will be conflict between two projects (i.e. both working on the same trigger).
• API Based Deployments – When code has successfully passed the necessary gates it can be migrated to the next environment on the release train.  Use change sets or ANT as well as detailed documentation.

#6 – System Test

This environment should be reflective of your desired production state. This is where I would recommend acceptance testing.  You should be able to conduct end-to-end business processes in this environment.  I typically would use this environment for my Full Copy sandbox as it will change less often than other environments.  Load testing and performance testing can also be validated in this environment due to the larger database size.

• Refreshes – I would refresh the system test sandbox as often as is practical.  Ideally you would refresh this every 29 days.
• Manual Changes (Configuration and Data) – If you will not refresh your sandbox then expect a much higher level of manual maintenance.  All production changes must be pushed into this environment one way or another.  Data that comes into the full copy sandbox may need to be changed to support testing, integration, or even privacy requirements.  Tooling and automated scripts come in handy here.
• API Based Deployments – When code has successfully passed the necessary gates it is ready to move to production.  But not QUITE ready… you must first complete a staged deployment.

#7 – Stage

This environment’s sole purpose is to ensure your deployments to production will succeed.  I would definitely not use a full sandbox here.  In fact I would refresh this environment prior to EACH production deployment.  That way you know you have the most up to date configuration from production.  I hope you followed my advice and used change sets or ANT throughout the migrations.  That means you have a comprehensive deployment manifest.  I also hope you followed my advice and kept detailed documentation for all manual changes.  That means you have a comprehensive set of deployment instructions.  After deploying to stage you should conduct smoke testing to ensure your deployment was successful.

• Refreshes – Refresh often.  Enough said.
• Manual Changes (Configuration and Data) – Theoretically you would not need to maintain any manual changes to this environment as they would all be brought over via the refresh.  However some manual steps to provision the environment will still be necessary, especially data related to any smoke testing.
• API Based Deployments – You can choose whether you want to deploy to Production from Stage or System Test.  However I am recommending only using Stage for practicing your deployment, in which case you could tear down the environment immediately upon validating your mock deployment.

If your deployment was successful then you have a very high assurance that your production deployment will also be successful.  Repeat the steps used in the stage deployment during the production deployment.

#8 – Training Environment(s)

Training is a difficult issue due to the nature of environment setup and tear down.  It is also difficult to determine whether to have the code that is ABOUT to go live versus the code that HAS ALREADY gone live.  So you have a few choices.  You can decide on one, multiple, or all of the following:

1. Migrate from System Test into a “Pre-Release Training Environment”.  Similar to a stage deployment but this environment would be persisted and will need a much more thourough data set to support training.
2. Refresh from Production into a “Post-Release Training Environment”.  This will ensure you have the latest and greatest metadata to support training.  You will still need to maintain data in this environment.
3. Train directly in production.  Use mock data that is recognized (and therefore ignored) enterprise wide.  This sounds scary but I’ve seen it work very well.  It also lowers your environment management costs.
4. Create training applications or videos.  You can create training videos or even interactive HTML5 applications that allow users to observe small business processes.  This can often be done much cheaper than maintaining live training environments.

#9 – Pfix (Production Fix)

No matter what you do, someday a Sev 1 bug will reveal itself.  So where should you make the fix and how do you get it deployed to production as quickly as possible?  My recommendation in these situations is to generate a new sandbox only to respond to the Sev 1 defect.  The configuration or code can be fixed, immediately tested, and immediately deployed to Production.  This should ONLY be done under dire circumstances as you are bypassing many of the controls I have outlined in this article.  Just make sure to reapply the fix into your sub-prod environments as necessary.

Enough Already!

As you can see there is quite a bit of activity that needs to take place in order to orchestrate a pristine environment plan.  And this article only covers migration actives, not even production support issues like data backup, archive, etc.  I hope it is clear by now that I HIGHLY recommend hiring a full-time Salesforce.com Environment Manager to plan, execute, and monitor all of the items outlined above.  An Environment Manger would be similar to a System Administrator – however their focus is very different.  A Salesforce system administrator is typically focused on users.  An Environment Manager would only be focused on the technical infrastructure.

When consulting with enterprise customers I often encounter issues like “how to maintain sandboxes” and “how to ensure smooth production deployments”.  And typically none of these customers have a dedicated Environment Manager who is following a robust strategy and executing these detailed tactics.  An investment in a dedicated Environment Manager will allow your company to scale Salesforce1 must more effectively.

Summary

Environment management is one of the most difficult and under-realized aspects on enterprise projects.  Companies often under estimate the amount of work necessary to orchestrate the movement of data and metadata throughout the environments.  The right strategy, the correct resources (human and technical), and effective processes can radically accelerate and improve your consumption of the Salesforce1 Platform.  You can harness the agility of Salesforce with the predictability of enterprise class deployments.  If you are looking for help on getting started, consider obtaining the help of a Salesforce.com Certified Technical Architect to help your company define and execute upon an Environment Management strategy.