LOGIN | REGISTER NOW | SUBSCRIBE
 
 
News on Monday
more>>
SharePoint Tech Report
more>>


   

 
 
Download Current Issue
ISSUE 3/15/2010 PDF

Need Back Issues?
DOWNLOAD HERE

Receive the print Edition?


 
blogs tab
Windows Phone 7 will not support HTML 5
Windows Phone 7 will not support HTML 5.
03/15/2010 05:51 PM EST

ASP.NET MVC 2 Ships
ASP.NET MVC 2 has shipped.
03/12/2010 10:26 AM EST

Microsoft plans 'open' Silverlight analytics framework
Microsoft is going to announce a multipurpose analytics framework for Silverlight at MIX.
03/11/2010 09:51 AM EST

 

Events calendar tab
SHARE
3/14/2010 to 3/18/2010
Seattle, Wa.
SHARE

Cloud Connect
3/15/2010 to 3/18/2010
Santa Clara, Calif.
TechWeb

Microsoft MIX10
3/15/2010 to 3/17/2010
Las Vegas
Microsoft

SharePointPro Summit & Expo
3/16/2010 to 3/19/2010
Las Vegas
Penton Media

TheServerSide Java Symposium
3/17/2010 to 3/19/2010
Las Vegas
TechTarget


 
print Printable version 
Most Read Latest News Blog Resources
Mono brings .NET to Android
At Microsoft MIX, Mono will unveil the ability to port .NET applications to the Android op...

Guest View: The half-agile path leads nowhere
While businesses are taking on agile and adopting its practices, an incomplete implementat...

Even with its success, .NET causes some consternation
Despite its popularity, the developer community believes that .NET isn't being fully serve...

Digg!  Digg
Reddit  Reddit
Slashdot  Slashdot
Share on facebook  Facebook
Share on Twitter  Twitter

Blind Spots and Frequent Testing


By Andrew Binstock
January 15, 2005 — 
The requirement that submissions to open-source projects be accompanied by unit tests reflects the trend among developers to perform frequent unit testing (FUT). I coin this acronym because I need to capture this comparatively new activity. It’s one step down from Extreme Programming’s Test-Driven Development (TDD) in that the code is written first (then immediately tested). My perception is that many exponents of TDD eventually drift into FUT. The tendency to write code first is both natural and expedient.

The trouble with FUT is that the tests and code are developed nearly in tandem, and so they fit each other well. Tests frequently focus on the main use case and then add the occasional edge value to make sure maximum, minimum and special values do not break the code. Not surprisingly, the test cases reflect the code already in the routine. The result is that the tests simply reinforce the perception that the code is functionally correct, without in fact really exercising potential weaknesses. I’ll come back to this in a moment.

A second problem, which needs attention if FUT ever is to do more than baseline validation, is to design frequent larger-than-unit tests. (I’ll avoid a new acronym here.) Unit tests are very much a bottom-up experience, designed and performed at a low level. They inspire a false confidence that the functionally correct units work together properly. This higher level of correctness, fortunately, is covered by standard testing tools that can drive cross-unit interfaces, simulate transactions and test systems under load. The trouble is, these tests are not run as frequently as unit tests because they take longer to develop and run—meaning that feedback to the developer is never immediate and often distant from the coding work by days, weeks, even months.

These larger tests are typically written by QA and test engineers rather than the developers themselves, and as a result they are less prone to testing what is already known to work. Testers can see in the code things that developers miss.

Between these two levels exists the code review, which generally comprises more than a single unit and is performed by other developers. Code reviews are great for enforcing standards, locating what Extreme Programming calls “smells,” and making sure that project coding is on track.

The problem is that reviewers tend to share the same blindness that affects the original developers, whether it comes in the form of skimming code constructs that are familiar, or similarly judging whether a given routine will ever be executed with, say, pathological data. These assumptions can be deadly wrong.

The explosion of the Ariane 5 rocket in 1996 was traced to an integer-conversion error. None of the managers who signed off on the code expected values would reach a level sufficient to cause an exception. When the values did, the code blew up, taking the rocket with it.

This well-known example should not be dismissed as a one-off example of human error. Programmer blind spots are everywhere. Consider one that amazes me, mostly because it is so conspicuous and never, ever handled: the command-line switch. Under Unix, Linux and Java, the default command-line switch begins with a hyphen. Under all three environments, filenames that start with a hyphen are also legal. Yet I have never seen a parser that checks a command-line argument that begins with a hyphen to make sure it’s not a parameter.

Even enterprise-safe JVMs screw this up. Try running Java on a file called –help.class; it won’t work. If you think filenames that begin with a hyphen are so rare as to not be worth testing, I would suggest you read the Ariane report, which is widely available on the Web.

FUT cannot deliver more reliable code until developers change their approach to testing and validation. The operating principle today is: “The code works, and I’ll write tests to prove it.” The principle must become: “The code does not work under all circumstances, so what test can I write to break it?”

Today, only one group of programmers I know of does this: the folk who worry about security. They correctly view all forms of input data as malicious and pore over programs to find ways in which malicious data could unhinge code. As a result, their testing is completely different: They look for perverse cases, document areas that could be weakened, and try to dream up ways in which their programs can fail.

Until developers and testers become more oriented this way, overall code quality and reliability will still fall short of what we should be delivering with FUT. z

Andrew Binstock is the principal analyst at Pacific Data Works LLC.


Share this link: http://www.sdtimes.com/link/28368
 

Add comment


Name*
Email*  
Country     


  • Comment
  • Preview
Loading