Integration Watch: Continuous testing cometh

By Andrew Binstock

April 15, 2009 —

A core practice of the agile movement is to release often. The goal of many small releases is to provide developers with feedback early and often on what they’re doing. It also allows users to make changes along the way that enable the project to completely fill its requirements and, especially, user and customer expectations.

As the practice of frequent releases proved to be a capable means of improving products by getting frequent input, agile practitioners realized it had a similar effect on developers within the development team. That is to say, if developers completed their coding cycle quickly and began the non-coding activities promptly, they could get near-immediate feedback on the changes they just made.

When thought about in this context, unit testing is a formidable tool for providing just such feedback. For example: You code, run tests, figure out what you broke, and decide how to respond to the tests’ feedback. You might suddenly discover that if you continue down the path you were innocently traveling, you will unhinge a key module that would take far too long to fix. You discreetly back up (you can tell how far to back up by the test feedback), and you devise a different approach that skirts the problem the unit tests have just flagged. This scenario makes good use of tests. And other similar examples all suggest that the sooner you run tests after you code, the earlier you can detect problems and make the necessary adjustments.

The recognition that the earlier you test your code, the easier it is to fix, spread out into new pastures a few years ago—namely, in builds. As all developers know, it’s not enough that code passes tests, it must also not break the build. To provide quick feedback on build-breaking check-ins, the practice of continuous integration emerged a few years ago. As I have discussed in previous columns, continuous integration software monitors a source-code repository and starts up a new build of the project every time code is checked in. This enables developers to know within a few hours whether they have broken the build and how.

Prior to continuous integration, a project might go a full day without a full build, or sometimes longer. So when the build broke, the ability to recall the motivation for specific changes is limited by the delay. But if the feedback can be given within a few hours from the code check-in, the developer can more likely zoom in on the defective code and fix it quickly.

A new trend called continuous testing (CT) is now emerging. It consists of running unit tests in the background within the IDE while the developer is coding. Depending on the product, there are various triggers for kicking off the test suite. Compiling the code is a common choice. The products that perform CT typically show a discreet red/green bar inside the IDE to which the developer can respond as needs and preferences dictate.

The idea, once again, is to test the new code against preceding tests, be they unit tests or characterization tests, to monitor the scope of changes. By running the tests often, the feedback loop is much shorter and the developer can make better decisions along the way. According to research by David Saff of MIT, who originated the concept of CT in roughly 2003, developers who used CT consistently delivered changes on schedule. This, I believe, is because they were never blindsided by unexpected consequences of their changes.

There are other advantages to CT: If the traditional non-TDD model is followed, code is written, then tests are run, then tests for the new changes are written and run. The cycle of running tests can take a while on large projects, meaning that developers will get up and do other things while the tests run and will frequently be lured into the growing distraction gap caused by tools (e-mail, instant messaging, Twitter, etc.) that clamor for attention. By getting the feedback right away, there is no pause. Moreover, CT does away with the tendency to write huge amounts of code before running tests.

Today, there are two principal CT products in the Java space: Ben Rady’s Infinitest and Kent Beck’s JUnitMax , a subscription-based product that’s still in beta. Both products are available as Eclipse plug-ins, and Infinitest is available for other IDEs, notably IntelliJ from JetBrains. For Ruby programmers, ZenTest is the only CT tool I know of.

In a future column, I expect to look at these products in greater detail.

Andrew Binstock is the principal analyst at Pacific Data Works. Read his blog at binstock.blogspot.com.

Related Search Term(s): testing

Share this link: http://www.sdtimes.com/link/33398

Integration Watch: A new approach to testing: Crowdsourcing Gathering many testers together is uTest's model, and it is increasingly becoming a viable and sound option for developers

ThoughtWorks puts new twist on testing The latest release of this test automation software supports behavior-driven development and allows for common languages

Letters to the Editors: An alternative development testing method A reader points out a perceived flaw in functional and traditional testing methodologies, and posits his own technique

Add comment

Name*
Email*
Country

Comment
Preview

News on Monday
more>>
SharePoint Tech Report
more>>

Download Current Issue
ISSUE 3/15/2010 PDF

Need Back Issues?
DOWNLOAD HERE

Receive the print Edition?

Google Code turns 5
Google Code Turns 5, and adds a Paxos Algorithm to make the system more stable and reliable.
03/17/2010 11:16 AM EST

Test your Visual Studio 2010 know-how
Microsoft is offering free beta certification exams for Visual Studio 2010.
03/17/2010 11:08 AM EST

Microsoft lifts the hood on IE9
Microsoft is previewing IE9.
03/16/2010 01:10 PM EST