I know you are all familiar with code coverage in Apex and the requirement to have 75% of your code covered by automated tests.  However, I have heard many questions regarding the different code coverage numbers you see in your orgs.  This post should help clear up some of the confusion in how the various numbers are calculated.

We Store Your Code Coverage

The first thing to know is that the system stores data from your latest test runs.  Each time you run the tests in a class, an entry is made in the code coverage table for each class you touch with those tests.  In addition, the lines you executed in those classes by those tests are recorded.

The coverage data allows you to know what the code coverage is in absence of changes to your classes.  Since you are likely changing things a little at a time between test runs, this is a pretty good approximation of the current code coverage in the system, although it will be slightly out of phase based on any recent modifications.

How That Stored Coverage Data Gets Used

When you click on “Calculate my org’s code coverage”, the result is based on the code coverage table.  Clicking the link is not doing a re-run of all your tests, nor is it doing any static analysis to figure out how much code would be covered.  It is merely providing an estimate of the coverage based on the most recent runs of each Apex test.

The code coverage table is used to display those blue and red highlights showing you which lines in a class are covered and which are not.

You may have noticed the numerator/denominator lines in the Apex Classes list page, which show how many lines are covered based on how many lines could be covered.  These values are calculated based on the code coverage table, too.

The Stored Code Coverage Data is not Necessarily Up To Date

The code coverage data is stored when your tests finish running.  At that point in time, you can be pretty sure that they’re accurate.  After that, things have the potential to lose accuracy.

Each record in the code coverage table is a junction object between the test being run and the class being covered.  Each record is valid until either the class with the test or the class being covered are modified.  If either parent class is altered, the code coverage record is deleted, removing the knowledge of the last run of that test.

In the case of the test being modified, whatever was covered in the past might no longer be covered, so the row is no longer valid.  For a class being modified, the lines within the class have changed, and the coverage is stored by line number.  As you can infer, if you add 15 lines of Apex to a class, these line numbers will no longer match for lines that have been shifted down several lines, and we are not in the business of guessing which new rows would have been covered by a prior test.

Fortunately, all you need to do to get the table refreshed with the most recent information is to run any relevant tests.

Based on this, the three areas where the coverage table data is used are approximations, and not guaranteed to be accurate, unless you have not changed anything since the last run of tests.  The org code coverage total could be wildly low, as all rows for changed classes are removed.  The per-class coverage could be low if relevant test classes have changed, and there might be inaccurate red highlighting for the same reason.  Please keep this in mind when you are looking at these numbers.

When Is Code Coverage Calculated?

There are two key times at which tests are run: when you explicitly execute tests, and when you do a deployment.  Coverage is calculated both times, but the two are different.

Explicit execution is typically invoked through the test user interface (preferably using the developer console, but also in the Setup -> Develop -> Apex Text Execution page).  This invocation will queue up tests to run asynchronously, with the tools providing a window into the results.  When tests finish running, the code coverage tables are updated with the results of the test run.

Tests are automatically executed when doing a deployment.  The requirement is that 75% of your overall Apex is covered by tests, so at deployment time the coverage is calculated for the tests being run.  The results of this calculation is NOT stored in the code coverage tables.

The reason code coverage results are not stored on deployment is to support rollback.  The coverage is stored as data in its own transaction.  When you run tests in the developer console, you are not persisting anything to the data store besides these records.  With a deployment, you are modifying metadata in addition to running tests.  If a deployment fails, the entire deployment is rolled back.  If any coverage data were saved in a separate transaction, it could be pointing to lines that never existed and to classes that were never committed.  This would be bad!  Thus, the test run done at deployment will not modify the stored coverage records.

Running “All” Tests

Another notable difference between these test runs has to do with managed package tests.  By default, a deployment will only run all unmanaged tests, i.e. tests you have created in your org.  (This can be overridden by setting the “runAllTests” flag to true on deployment.)  If you click “Run All Tests”, or select all tests in the user interface, you will be running both your unmanaged tests AND any managed tests that exist in your org.

Managed package class coverage is ignored for overall calculation.  However, managed tests can execute your trigger logic, and might cover portions of your code that were not covered elsewhere.  This would show up in the approximate coverage statistics, but would not be included upon deployment.

When you click the “Run All Tests” button in the setup UI, you are enqueuing all tests, managed and unmanaged, to be executed.  That button doesn’t offer you flexibility to choose; the test user interface in the Developer Console and the Apex Test Execution page allow you much more control over what tests are being run.  You can select by namespace, so it’s very easy to ignore managed tests (since they will be in a different namespace than your unmanaged tests).

Please Note:  The “Run All Tests” button (and the “Run Test” button on an individual class page) currently runs the tests synchronously, showing you a blank-ish screen while tests are run, and filling the screen with results when the tests complete.  As of Spring ’13, these buttons will enqueue the tests to run asynchronously, as is done today in the Developer Console and Apex Test Execution UI.  This allows parallel execution, avoids the blank-ish screen, and gives you a better window into test progress.  You can start using these improved user interfaces today – no need to wait until Spring ’13 for us to nudge you there!

Aggregated Code Coverage Statistics

Some of you in the developer community have written an immense number of tests, for which I salute you.  This achievement, however, can sometimes make storing code coverage slow; this can be especially noticeable when you do a run-all-tests, which invalidates and recreates all of your code coverage junction objects.  There is a way to get around this problem – aggregated code coverage statistics.

On the Apex Test Execution page, there is an “Options…” button.  This button reveals an option titled “Store Only Aggregated Code Coverage”.  Aggregated statistics will store the total number of lines covered for any given class.  It will not, however, store the effects of each individual test class which provides any portion of that coverage.  The significantly reduces the potential number of code coverage records being stored, and can speed up the performance of test runs.

The drawback to the aggregated option is twofold.  First, it will only store the most recent run of tests, regardless of how many tests were run.  If a single test class is executed, all you will find in the table is the result of that one test.  Second, if any test class is modified, the entire code coverage will be invalidated and your org will appear to have no coverage.  If you wish to have the per-test coverage data, you will need to keep this option unselected.

Approximate Coverage Versus Actual Coverage

Based on all of the ways in which coverage can be stored and invalidated, the stored coverage numbers should all be seen as approximates.  The only way to get the true coverage for your org is to run all unmanaged tests, or to run a deployment.

Even the removal of coverage data does not always give you a clean slate.  If your test calls a method class A which in turn calls a method in class B, running this test creates a record for both class A and class B.  If you change the method in class A to remove the call to class B, the class A coverage will be removed (since the class is changed) but the class B coverage will remain in place (since the class did not change and the system does not try to run a static analysis on the possible code path implications of the class A change).

You will sometimes see distinctions between the numbers shown to you in the user interface and the numbers seen on a deployment, and hopefully this has helped you understand why you might see a difference.  Now you know more, and you are a stronger developer.  Happy coding!

Bookmark the permalink. Trackbacks are closed, but you can post a comment.
  • http://siddheshkabe.co.in Siddhesh Kabe

    If we are doing iterative deployment and few classes are already in production. Will the code coverage of those existing classes be considered while pushing the new classes? For aggregate 75% coverage

    • Naveen Gabrani

      When a deployment is being done, all the test cases will be run on Production. Code coverage will be computed based upon all the classes present in production. It does not matter which file is being deployed.

  • Vardhan Gupta

    One thing I noticed that if multiple lines of code was written all on the same line, then if your test-case covered only first part of the logic, it counts it as code coverage for the entire logic.

    for e.g. For the below statement:
    if (condition) {statement 1; statement 2; } else {statement 3; statement 4;})

    If your test class provided coverage only for the IF block and did not check the ELSE condition, just by virtue of the fact that the entire code is listed on the same line, the entire code is covered.

    Has others experienced the same? If true, folks may use this technique as shortcuts to write meaningful tests.

    • Naveen Gabrani

      Yes that is correct. If multiple lines of code is written on the same line, and if one of them gets executed, and others are not, that line will be considered as executed.

    • http://twitter.com/FishOfPrey Daniel Ballinger

      Yes, I’ve come across the same issue. The code coverage only considers lines, not code branches that may be within the lines.

      http://www.fishofprey.com/2011/05/avoid-single-line-salesforce-apex-if.html

  • Scott Hemmeter

    For some time now, the code coverage in our main developer org has been all messed up. For example, I just ran all tests. All passed. I look at a class that has 58% coverage. The lines it says are not covered are comment lines , blank lines, lines that are part of a multi-line command (e.g. if I put a SOQL statement on multiple lines, it counts some as not covered). If you look through the lines identified as not covered, I see 4 lines that are actual code. The real code coverage should be like 98%.

    These bad calculations have never stopped us from deploying so we never said anything and figured it’s a big and we’d deal with it some time later.

    Something changed in the last 3-4 months that caused this to start. The same very poor code coverage stats come through in Eclipse too. We’ll run tests and the code coverage won’t even list classes that execute.

    I’ve lost faith in the test coverage stats for the past 3-4 months.

  • Scott Hemmeter

    More on this. I did some looking into. If I look into the detailed code coverage, it’s all messed up. If I use the UI and choose the main test class that should cover my class, it looks accurate only showing a few lines as not covered. When all my tests are taken into account, the actual code coverage for my class is way way off.

    • http://twitter.com/rich_unger Rich Unger

      Strange. Is anyone else out there seeing this?

  • Scott Hemmeter

    I”m back and the issue was resolved. For some reason, our code coverage tables were not being rewritten when we executed tests. This is why we were seeing bad lines showing as not covered. It was old data. The alignment of what was in the code coverage tables was not in sync with our actual code.

    Running a single test class showed good coverage on a class, but running all never had the right stats. And the code coverage calculations we see looking at the list of Classes was never right.

    We flushed the code coverage tables and reran everything and it got back in sync. Seems like things are better now. I assume Eclipse will be better too.

  • http://www.facebook.com/sandeep.k.singhal Sandeep Kr Singhal

    Hi Josh,

    Thank you so much for providing this deep analysis of test execution. It is very helpful. could you please let us know why test code coverage from UI and from eclipse comes different % many time. ( when run only single test class)

    -Sandeep