I know you are all familiar with code coverage in Apex and the requirement to have 75% of your code covered by automated tests. However, I have heard many questions regarding the different code coverage numbers you see in your orgs. This post should help clear up some of the confusion in how the various numbers are calculated.
We Store Your Code Coverage
The first thing to know is that the system stores data from your latest test runs. Each time you run the tests in a class, an entry is made in the code coverage table for each class you touch with those tests. In addition, the lines you executed in those classes by those tests are recorded.
The coverage data allows you to know what the code coverage is in absence of changes to your classes. Since you are likely changing things a little at a time between test runs, this is a pretty good approximation of the current code coverage in the system, although it will be slightly out of phase based on any recent modifications.
How That Stored Coverage Data Gets Used
When you click on “Calculate my org’s code coverage”, the result is based on the code coverage table. Clicking the link is not doing a re-run of all your tests, nor is it doing any static analysis to figure out how much code would be covered. It is merely providing an estimate of the coverage based on the most recent runs of each Apex test.
The code coverage table is used to display those blue and red highlights showing you which lines in a class are covered and which are not.
You may have noticed the numerator/denominator lines in the Apex Classes list page, which show how many lines are covered based on how many lines could be covered. These values are calculated based on the code coverage table, too.
The Stored Code Coverage Data is not Necessarily Up To Date
The code coverage data is stored when your tests finish running. At that point in time, you can be pretty sure that they’re accurate. After that, things have the potential to lose accuracy.
Each record in the code coverage table is a junction object between the test being run and the class being covered. Each record is valid until either the class with the test or the class being covered are modified. If either parent class is altered, the code coverage record is deleted, removing the knowledge of the last run of that test.
In the case of the test being modified, whatever was covered in the past might no longer be covered, so the row is no longer valid. For a class being modified, the lines within the class have changed, and the coverage is stored by line number. As you can infer, if you add 15 lines of Apex to a class, these line numbers will no longer match for lines that have been shifted down several lines, and we are not in the business of guessing which new rows would have been covered by a prior test.
Fortunately, all you need to do to get the table refreshed with the most recent information is to run any relevant tests.
Based on this, the three areas where the coverage table data is used are approximations, and not guaranteed to be accurate, unless you have not changed anything since the last run of tests. The org code coverage total could be wildly low, as all rows for changed classes are removed. The per-class coverage could be low if relevant test classes have changed, and there might be inaccurate red highlighting for the same reason. Please keep this in mind when you are looking at these numbers.
When Is Code Coverage Calculated?
There are two key times at which tests are run: when you explicitly execute tests, and when you do a deployment. Coverage is calculated both times, but the two are different.
Explicit execution is typically invoked through the test user interface (preferably using the developer console, but also in the Setup -> Develop -> Apex Text Execution page). This invocation will queue up tests to run asynchronously, with the tools providing a window into the results. When tests finish running, the code coverage tables are updated with the results of the test run.
Tests are automatically executed when doing a deployment. The requirement is that 75% of your overall Apex is covered by tests, so at deployment time the coverage is calculated for the tests being run. The results of this calculation is NOT stored in the code coverage tables.
The reason code coverage results are not stored on deployment is to support rollback. The coverage is stored as data in its own transaction. When you run tests in the developer console, you are not persisting anything to the data store besides these records. With a deployment, you are modifying metadata in addition to running tests. If a deployment fails, the entire deployment is rolled back. If any coverage data were saved in a separate transaction, it could be pointing to lines that never existed and to classes that were never committed. This would be bad! Thus, the test run done at deployment will not modify the stored coverage records.
Running “All” Tests
Another notable difference between these test runs has to do with managed package tests. By default, a deployment will only run all unmanaged tests, i.e. tests you have created in your org. (This can be overridden by setting the “runAllTests” flag to true on deployment.) If you click “Run All Tests”, or select all tests in the user interface, you will be running both your unmanaged tests AND any managed tests that exist in your org.
Managed package class coverage is ignored for overall calculation. However, managed tests can execute your trigger logic, and might cover portions of your code that were not covered elsewhere. This would show up in the approximate coverage statistics, but would not be included upon deployment.
When you click the “Run All Tests” button in the setup UI, you are enqueuing all tests, managed and unmanaged, to be executed. That button doesn’t offer you flexibility to choose; the test user interface in the Developer Console and the Apex Test Execution page allow you much more control over what tests are being run. You can select by namespace, so it’s very easy to ignore managed tests (since they will be in a different namespace than your unmanaged tests).
Please Note: The “Run All Tests” button (and the “Run Test” button on an individual class page) currently runs the tests synchronously, showing you a blank-ish screen while tests are run, and filling the screen with results when the tests complete. As of Spring ’13, these buttons will enqueue the tests to run asynchronously, as is done today in the Developer Console and Apex Test Execution UI. This allows parallel execution, avoids the blank-ish screen, and gives you a better window into test progress. You can start using these improved user interfaces today – no need to wait until Spring ’13 for us to nudge you there!
Aggregated Code Coverage Statistics
Some of you in the developer community have written an immense number of tests, for which I salute you. This achievement, however, can sometimes make storing code coverage slow; this can be especially noticeable when you do a run-all-tests, which invalidates and recreates all of your code coverage junction objects. There is a way to get around this problem – aggregated code coverage statistics.
On the Apex Test Execution page, there is an “Options…” button. This button reveals an option titled “Store Only Aggregated Code Coverage”. Aggregated statistics will store the total number of lines covered for any given class. It will not, however, store the effects of each individual test class which provides any portion of that coverage. The significantly reduces the potential number of code coverage records being stored, and can speed up the performance of test runs.
The drawback to the aggregated option is twofold. First, it will only store the most recent run of tests, regardless of how many tests were run. If a single test class is executed, all you will find in the table is the result of that one test. Second, if any test class is modified, the entire code coverage will be invalidated and your org will appear to have no coverage. If you wish to have the per-test coverage data, you will need to keep this option unselected.
Approximate Coverage Versus Actual Coverage
Based on all of the ways in which coverage can be stored and invalidated, the stored coverage numbers should all be seen as approximates. The only way to get the true coverage for your org is to run all unmanaged tests, or to run a deployment.
Even the removal of coverage data does not always give you a clean slate. If your test calls a method class A which in turn calls a method in class B, running this test creates a record for both class A and class B. If you change the method in class A to remove the call to class B, the class A coverage will be removed (since the class is changed) but the class B coverage will remain in place (since the class did not change and the system does not try to run a static analysis on the possible code path implications of the class A change).
You will sometimes see distinctions between the numbers shown to you in the user interface and the numbers seen on a deployment, and hopefully this has helped you understand why you might see a difference. Now you know more, and you are a stronger developer. Happy coding!