Incremental code coverage as a debugging tool

by Kaitlin Duck Sherwood, March 12th, 2012

(See also this article’s translation to Serbo-Croatian language by Vera Djuraskovic at http://science.webhostinggeeks.com/tasktop-blog)

I joined Tasktop in part because I share the goal of increasing programmer productivity, especially by filtering out unimportant information.  I also liked how Tasktop is committed to being involved with and connected to the broader Eclipse community.

In this spirit, I suggested a feature to the EclEmma project: letting developers create and view incremental code coverage results.  This would let developers see a much smaller, but more relevant, set of classes and methods which they could then investigate.

It had bothered me how difficult it was to find where things happen in code, especially in large, unfamiliar code bases.  Yes, you can step through the code, but sometimes you have to step for a long time.  This is boring and tedious, and frequently you step one step too far and overshoot the place you wanted to see — losing the information about the values of the variables at the point you cared about.  If an asynchronous process gets spawned as the result of a Listener attached to a GUI element, it is nearly impossible to step through.

You can search for text on or around the GUI element to help you find where the actions related to that element are processed, but sometimes this isn’t practical.  Sometimes the GUI element has text that is so common that it is impractical to search for it, like “Finish” or “Next”.  Sometimes the GUI element doesn’t have text associated with it, like a button with a picture on it (and no tooltip).

In practice, I have observed that people usually guess at what words might be included in the class or method names, and then when they think they are close, simulate stepping through code by reading through it and making informed guesses about where the execution flow will go.  Unfortunately, frequently they guess wrong, usually in the choice of a starting point or the value of an if-condition.

Another particularly pernicious mistake is not realizing that you are tracing through a superclass of the class that is actually executed.  If you Command/control-click on a method name, Eclipse will preferentially take you to the implementation of that method in the same class; this means that if you ever trace into a superclass, Eclipse will tend to keep you in the superclass; realizing that you need to go back to the subclass is not always obvious.

I thus suggested to the EclEmma code coverage team that they add a feature to the code coverage tool to let users start and stop coverage so that they can see which code was executed for specific short periods of the execution.

With the 2.1 release of EclEmma, the EclEmma team has implemented incremental code coverage — a very useful feature!

How to use it

First, install EclEmma.

Next, go to Preferences > Java-Code Coverage, and check the “Reset execution data on dump”.

Open a Coverage view.  In the Coverage toolbar, there is a “Dump Execution Data” button.

Pressing the “Dump” button will now display coverage and reset the code coverage results.  Thus, if you press the dump button right *before* you do the action you are interested in (e.g. pressing a certain button), and then again right after you do the action, then the code coverage results will show only the exact classes and methods that were executed in response to that action.  Among other things, this means you won’t get misled to look in the superclass instead of the appropriate class.

If you need finer-grained information about which code executed, EclEmma colours your lines of code based on whether they were executed in that interval: green if they were fully executed, red if they were not executed at all, and yellow if they were partially executed (if, for example, the line uses a “?” ternary operator, as in the “drightSide” assignment below).

By default, EclEmma only shows coverage information for your code, not for all the libraries you bring in.  To change this, uncheck “Source folders only” in Preferences > Java > Code Coverage.

Note: while for code coverage, you probably want to see the coverage results for all classes and methods, when using incremental code coverage to locate places in code, you should select “Hide unused elements” from the Coverage toolbar’s drop-down menu.

Incremental code coverage is a very powerful technique: instead of wandering through thousands of classes and methods to find the handful of classes and methods that are interesting, you can spend a few minutes to get their names directly.

I do need to give a slight caveat: exceptions interfere with the code coverage instrumentation, interrupting the marking of that branch of code as executed.  That is a known limitation with the way that code coverage is done.  Thus if your code uses exceptions a lot, EclEmma might incorrectly say that a branch of code was not executed when it was in fact executed.   However, if EclEmma tells you that code was executed, it really was executed.

Using EclEmma in conjunction with Tasktop Dev or Mylyn is an exciting prospect.  Mylyn and Tasktop Dev tell you what you (or someone else, if you are looking at their context) had looked at; EclEmma gives you hints on what you should look at. We have only just started thinking about how those two could be combined, but are excited by the possibilities.

For further information, see EclEmma, Tasktop Dev, or Mylyn.

Note: the screenshots used from the open-source Java GIS tool GpsPrune, isolating the action of showing the scale legend.

2 Responses to “Incremental code coverage as a debugging tool”

  1. Tomasz Zarna Says:

    Awesome, I’m updating my EclEmma right now!

  2. Daniel Says:

    I am wondering whether there is any code coverage tool which can find out the changed codes between two releases of my application and will show test coverage is 100% if the changed codes are all tested. Thanks a lot.

Leave a Reply