Using CruiseControl for Continuous Integration on Force.com

Abstract

Continuous integration refers to the practice of frequently integrating and testing code, rather than waiting until the end of a project or development cycle. The practice of continuous integration is a well established best practice that has been used with software development across multiple platforms and languages.

Implementing an automated continuous integration system usually involves a source code repository, used to store the code during team development, and a continuous integration tool that polls the repository for changes and automatically builds and deploys the code in a sandbox. This article shows you how to start using continuous integration on Force.com using the popular CruiseControl tool.

Introducing Continuous Integration

Consider a typical development cycle scenario at a fictitious company called Acme. Acme has multiple developers working on different parts of the project, with each of these developers working in their own Developer Sandbox or Developer Edition environments.

Before a release they integrate their code in a sandbox environment, perform their testing and finally promote it to a production. Acme has found that the integration phase always takes a lot longer than they expect—they encounter code that will not compile, unit tests that fail, and a misunderstanding of requirements across the team members.

To address this issue they decided not to wait until the end of the coding phase to do their integration, but rather do this on a more regular basis. They can designate someone to collect the code from the developers at the end of the day, integrate the code, build it, run the unit tests and notify the appropriate developer when a build or a unit test fails.

This obviously is a time consuming and expensive proposition. A better way is to automate this such that all the integration activities happen without any human intervention. There are two pieces to an automated solution:

  • A source code repository - which holds the source code for the product being built. Read Using Force.com with Subversion for Team Development to understand how to set up Subversion as a repository.
  • A continuous integration tool - this tool will regularly poll the source code system. If it detects any change like the addition of a new file or changes to an existing file, it will check out all files, build and run the code against a sandbox environment, and produce the results on a web based dashboard. It can also send an e-mail notification.

Pretty neat - huh? To accomplish this, we will use another open source tool called CruiseControl.

This process of frequently integrating and testing code rather than waiting until near the end of the project to assemble the separately developed pieces is called continuous integration. The practice of continuous integration is a well established best practice that has been used with software development across multiple platforms and languages. This article shows how you can incorporate continuous integration for your Force.com development.

CruiseControl with Force.com

CruiseControl is a continuous integration tool that allows you to define a custom build process that runs continuously at some frequency (once a minute, a hour day etc.). It includes dozens of plugins to integrate with source code control systems and notification systems like e-mail. It also provides a web dashboard for monitoring current and previous builds. Force.com, which includes the Migration Tool (for migrating metadata describing applications and configuration between environments) allows the integration of CruiseControl into a development project.

The rest of the tutorial illustrates the following:

  • The logical architecture of the integration - this will help you form a conceptual view of the solution.
  • Installing CruiseControl - the tool that does all the heavy lifting of continuous integration
  • Installing SvnAnt - we will be using Subversion as our source code repository. To enable the build script to retrieve files from Subversion, you will have to install an Ant task called SvnAnt. This section will show you how to install it as well as use this task in your build file.
  • Configuring CruiseControl - we will look at the configuration files and the build files and highlight a few important sections.
  • Running CruiseControl - enjoy the fruits of your labor! We start up CruiseControl and have it constantly monitor the source code repository and run the builds.

The tutorials end with some best practices and additional resources to get you started. This article builds on the Using Force.com with Subversion for Team Development article - please read it if you are not already familiar Subversion and team development.

Logical Architecture

Figure 1 illustrates a typical set up when using team development. Each developer has his own individual developer sandbox or developer environment. They check their code into a shared Subversion source code repository. CruiseControl is installed on a build machine, and it periodically checks the Subversion repository. If it detects any change in the repository, it retrieves the files and performs a build against an org by invoking the deploy task in the Force.com migration tool.

Team Development with CruiseControl

The results of the build can be seen through a web dashboard. Optionally, a notification in the form of an e-mail or IM can be sent on the build result to the team so that problems can be fixed quickly. Figure 2 shows a typical web dashboard.

Figure 2: CruiseControl's Web Dashboard

It is possible to not only track the history of builds but also drill into the details of each build if required, as shown in Figure 3.

Figure 3: Drilling into the details of a build

Let's get started! We're going to assume you know how to set up the team environment (it's documented in Using Force.com with Subversion for Team Development) and move straight on with installing the continuous integration tools.

Installing CruiseControl

Here's how to install CruiseControl:

  • Download CruiseControl.
  • Unzip it into a directory - we unzipped it under c:\software\CControl. It will create a directory called cruisecontrol-bin-2.8.2. If you have downloaded a different version, the last three digits may vary.
  • Create a directory under C:\software\CControl called builds - this directory will be used to hold the configuration files as well as the local working directory for our source files retrieved from Subversion.
  • Create additional directories called logs, logs\mileage and checkout to hold the log files and the local working directory.

The directory structure should look something like that in Figure 4. We will be using this in our configuration and build files, so if you have installed it into a different location, you would need to change the appropriate entries in those files to point to the correct locations on your machine.

Figure 4: CruiseControl directory structure

The CruiseControl distribution includes an Apache Ant distribution as well, which CruiseControl uses by default. You can either point it to your own Ant installation or copy the required custom Ant task JAR files into the lib directory of the Ant distribution. We will be taking the latter approach, so we will copy the Salesforce Migration tool jar (ant-salesforce.jar) and the SvnAnt JAR (see below) to the Ant distribution under CruiseControl - in our case it will be into the directory C:\software\CControl\cruisecontrol-bin-2.8.2\apache-ant-1.7.0\lib.

Installing SvnAnt

You need the ability to check the Subversion repository for any changes, and if changes are detected retrieve all the files. Since we are automating this entire process, we want to accomplish this via an Ant build script. For that you will need to download an open-source Ant task called SvnAnt.

After you download SvnAnt, unzip it and from the lib directory of the unzipped folder copy the three jar files - svnant.jar, svnClientAdapter.jar and svnjavahl.jar to your Ant installation's lib directory. In our case, it would be to C:\software\CControl\cruisecontrol-bin-2.8.2\apache-ant-1.7.0\lib.

We will come back to configuring SvnAnt in the next section.

Configuring CruiseControl

Create a file called config.xml in your builds directory - C:\software\CControl\builds. This is the main configuration file and the heart of your continuous integration system.

You can configure several projects within the file, we've just chose to do one which we've called mileage. You typically specify where the local working copy of the file is located, logs directory, how often it should be invoked, the Ant file to invoke etc. You can find complete documentation here. We have kept it bare bones for simplicity. Let us look at the schedule element of the mileage project:

<!--config.xml -->
<!-- cruisecontrol is the root element -->
<cruisecontrol>
  <!-- under it we can have several projects -->
  <project name="mileage" ... >
      
      <schedule interval="60">
	    <ant buildfile="cc-build.xml" target="build" />
       </schedule>

   </project>
</cruisecontrol>

Here we are calling the build target in an Ant build file called cc-build.xml. This is where we do our project specific work. This gets invoked every 60s - you can ofcourse modify it to suit your requirements. All the files referred to in this tutorial is available for you to download.

Remember the SvnAnt plugin we had downloaded and installed? The plugin is actually a custom ant task that helps us to checks the Subversion repository for changes and if it detects any retrieves the files to our build machine. To use this custom task from your Ant build files you would first need to load it in your cc-build.xml build file as shown below. Later on you will invoke the task.

<project name="cc-build" default="build" basedir=".">
	<!-- path to the svnant libraries. Usually they will be located in ANT_HOME/lib -->
	<property file="build.properties" /> 
  	<path id="project.classpath">
		<pathelement location="${svnjavahl.jar}" /> 
    		<pathelement location="${svnant.jar}" />
    		<pathelement location="${svnClientAdapter.jar}" />
  	</path>   
  	<!-- load the svn task -->
  	<taskdef resource="svntask.properties" classpathref="project.classpath"/>

       <!-- Rest of the build file -->

Let us now look at the build task in cc-build.xml. Here we first delete the entire trunk directory (we want a clean build), checkout all the files from Subversion and invoke a task called deployCode in another Ant file called build.xml. Though we could have kept everything in one file, this separation makes it easier to maintain our code and is a common CruiseControl convention as well.

<!-- cc-build.xml -->
<project ......>
        <target name="build">
		<delete dir="checkout/trunk" />
    		<svn username="guest" password="">
      		<checkout url="svn://localhost/mileage/trunk" destPath="checkout/trunk"  />
    		</svn>
                <!-- we call the deployCode task in another ant file here -->
		<ant antfile="build.xml" dir="." target="deployCode" />
  	</target>
</project>

The deployCode in build.xml is shown below. It invokes the deploy task of the migration tool. Make sure that the ant-salesforce.jar file is present in the Ant lib directory - C:\software\CControl\cruisecontrol-bin-2.8.2\apache-ant-1.7.0\lib in our case.

<!-- build.xml -->
<project ... >
   <property file="build.properties"/> 
 <target name="deployCode">
      <!-- Upload the contents of the "trunk" directory, CHANGE deployRoot TO MATCH YOUR environment-->
      <sf:deploy username="${sf.username}" password="${sf.password}" 
              serverurl="${sf.serverurl}"   deployRoot="checkout/trunk" logType="Debugonly">
		<!-- Additionally you can run specific unit tests here-->
                <runTest> ... </runTest>
	</sf:deploy>
    </target>
</project>

We are almost done. The last task remaining is to configure the dashboard configuration for CruiseControl. The configuration is contained in the dashboard-config.xml file, which should be present in the CControl\builds directory (or the directory from where you are going to invoke CruiseControl).

This file is pretty straight forward and we refer you to attached zip file that has all the configuration files. Finally, open up cruisecontrol.bat, which you can find in the root of your CruiseControl install (C:\software\CControl\cruisecontrol-bin-2.8.2 in our case), and define the dashboard.config setting like this: -Ddashboard.config=./dashboard-config.xml on the set EXEC line:

set EXEC=%JAVA_PATH% %CC_OPTS% -Djavax.management.builder.initial=mx4j.server.MX4JMBeanServerBuilder -Ddashboard.config=./dashboard-config.xml "-Djetty.logs=%JETTY_LOGS%" -jar "%LAUNCHER%" %* -jmxport 8000 -webport 8080 -rmiport 1099

Running CruiseControl

At this point you should have CruiseControl and SvnAnt installed and configured. We have defined the CruiseControl configuration in config.xml to check the Subversion repository every 60 seconds, and if any changes are detected it calls the Ant file cc-build.xml file that retrieves files from the subversion repository and calls build.xml, which compiles and deploys to a Salesforce org. We also defined the dashboard-config.xml file and modified cruisecontrol.bat to point to this file. We are now ready to run CruiseControl!

To get going, first, start up your Subversion repository if it's not already running. Since we are using the same repository we used in Using Force.com with Subversion for Team Development, we start up the repository as shown below from a command prompt:

> svnserve --daemon --root "c:\software\svn-test\repos"

Next, let us check that the build files do what we expect them to do. Let's invoke them manually as shown below. You should set JAVA_HOME to point to your Java SDK, and ANT_HOME environment variable to point to your ant installation - C:\software\CControl\cruisecontrol-bin-2.8.2\apache-ant-1.7.0 and your PATH should include the bin directory of the Ant installation.

> ant -f cc-build.xml build 

You should see output similar to that in Figure 5.

Figure 5: Running the build from the command line


Next let's start CruiseControl from the builds directory, which contains our configuration files:

> C:\software\CControl\cruisecontrol-bin-2.8.2\cruisecontrol 

You will see CruiseControl entering the build loop. If there is no change in the Subversion repository it goes to sleep for 60 seconds before checking again. Let's edit one of the source files and make a simple change, simulating what a developer may do and commit this change to Subversion:

> svn update MileageUtil.cls
> svn committ - m "testing CruiseControl" MileageUtil.cls 

You should now see something like Figure 6 in the CruiseControl command window.

Figure 6: An automatic build

You can also go to http://localhost:8080/dashboard/tab/builds to monitor the build and build history. This is the preferred method - replace localhost with the name of the build machine if you are accessing it remotely.

A Few Best Practices

We have sketched the basic structure to automate integrations and builds. You can take this as far as you want in terms of automation with more sophisticated build files and using CruiseControl's notification features, file parsing and other mechanisms. Here are a few general best practices to keep in mind:

  • Use a dedicated build machine. Since in the case of Force.com the heavy lifting of compiling happens on the Force.com platform in the cloud, this build machine can be a fairly cheap, lightweight machine. Having a dedicated machine makes it easy to schedule any number of builds without affecting any programmer's productivity.
  • Encourage the developers to check in their code as often as possible - the whole purpose of continuous integration is avoiding surprise at the end. At a minimum, developers should check in their code at least once a day.
  • Use a Test Driven Development methodology. This, along with continuous builds will help you catch bugs early when it is easier and cheaper to fix them.
  • A corollary to the above is to make these test meaningful by testing against data that simulates a production environment. So you can use either a full copy sandbox or a developer sandbox with good test data.
  • Though we have focused on code, you can use the Force.com Migration Tool for any metadata - we encourage you to do so since the same benefits apply to those integrating those changes as well on a continuous basis.

Summary

Continuous Integration is the practice of continuously or frequently integrating source code and other artifacts developed by various team members on a project. Doing so catches issues like misunderstanding of the requirements, coding bugs and other problems early, when it is easier and cheaper to fix. Automation is the key to reaping the benefits of continuous integration. This article shows how CruiseControl, along with Force.com Migration tool, can be used to automate this task, making it easy for developers to adopt this practice.

References

About the Author

Nick Simha is a Partner Enablement Manager and Platform Evangelist working with salesforce.com's consulting partners to help them build innovative cloud applications for our customers.