When the Server Crashes, Testing's Done
Stories Columns Opinions Resources
Preflight builds spread wings for smoother projects
Developers are increasingly turning to preflight builds, allowing them to experiment with ...
|
Coverity creates program to enforce code adherence
The Architecture Analyzer uses mapping technology from the company's Software DNA static a...
|
QCon 2008 features domain-driven development
This year's QCon invites speakers like Eric Evans and Dan North to talk about domain-drive...
|
.NET similarities prove golden for Silverlight
Microsoft has focused on making Silverlight 2 symmetric with the .NET platform, and that h...
|
SOA Watch: New economic realities
In the current economic downturn, agile programming and SOA are attractive options that bu...
|
Integration Watch: A new twist on threads
The key to raising the efficiency of multiprocessors is to shrink the overall workload by ...
|
Integration Watch: The Return of NetRexx?
Java scripting languages are seeing a surge in popularity, with NetRexx looking particular...
|
Windows & .NET Watch: Transaction crowd gets a boost
With multicore chips becoming the standard for processors, the need for a flexible, usable...
|
From the Editors: Election should shake up JCP
Rod Johnson has the right ideas for opening up the Java Community Process, and he may be a...
|
Letters to the Editor: Sun gives REST, SOAP choice
A reader takes issue with a headline on our story about Sun working with REST along with S...
|
Guest View: Be smart and lazy
The optimal solution for problems is the simplest one, so always aim to streamline your ap...
|
Zeichick's Take: From EXEC to EXEC 2 to REXX to NetRexx
Andrew Binstock's column last week, "The Return of NetRexx," brought back some fond memori...
|
Advanced Corda CenterView™ Data Visualization for the BusinessObjects™ Intelligence Platform
Corda Technologies presents a white paper on pervasive BI. The BusinessObjects business in...
|
From Mobile to SOA: A Guide for Optimized Application Deployment
Customer need has driven the emergence of multiple computing tiers. Today’s application d...
|
e-Kit: Web Application Security
Is your network secure? What about your web applications.
If IT security is your top p...
|
Practical tips for saving money on code maintenance
If software design is expensive, well, code maintenance is even more so. When you look...
|
By Edward J. Correia
June 3, 2008 —
The challenge for software teams—typically locked into time-boxed deployment schedules—is to squeeze as much testing into the available time as possible while maintaining high levels of quality. “When applied to load testing, fault injection tools allow testers to uncover more defects, especially gnarlier and more significant defects,” and do so in the same amount of time as other methods, says Pete Jenney, a 20-year software industry veteran now with risk analysis and mitigation consultancy Security Innovation. This can give managers the data needed to decide which features or functionality will ship on time and which will not.
Today’s complex applications are made of millions of lines of code from hundreds or thousands of libraries assembled from scores of software makers, and none of it guarantees quality or stability, Jenney asserts. He has observed that to compensate, developers maintain a certain belief system that allows them to use unfamiliar libraries while keeping their sanity intact.
This system assumes that:
• Resources consumed by their applications are always available and correct.
• Their applications own all the resources on the system.
• Nobody can monkey with the system resources they consume.
As we know, none of these assumptions are particularly safe, and almost certainly not at the same time. What’s worse, Jenney says, is that “as managers, we enable these belief systems by imposing unrealistic time constraints, staffing and feature loads on the development team. Experiencing ‘urgency’ in the development process forces developers to take shortcuts and make assumptions regarding how an application will be used when deployed.” This leads to simple code problems that can have dramatic effects, as seen in the following code:
try
{
newThingy = new criticalResource();
}
catch
{
// TODO: figure out what to do if we get here
}
“It is this belief system and the urgency imposed by management that leads to fragile and insecure applications, and it’s not likely that anything will come along that will relieve the market pressures that lead to it,” Jenney says.
One solution is to improve the depth of testing applied to an application without increasing the duration of the testing process. According to Jenney, the simplest way to do this is with fault injection.
Fault injection is a method of forcing applications to execute code paths meant to handle error recovery, and increasing an application’s test surface, explains Jenney. “By simulating the conditions that one would expect to force errors—such as no disk space, no memory, missing DLLs, corrupted data files or broken network controllers—defects that would be missed in normal testing are exposed. By applying fault injection to the normal automated functional testing process, it is possible to uncover not only more defects, but more interesting ones, and typically those which are much harder to find.”
What’s more, the technique exposes defects not only in the team’s own code, but in libraries and resources consumed by the application. “This destroys the belief system upon which development depends,” Jenney states, and “encourages better code and more appropriate management behavior.”
The benefits for functional testing should be obvious, but the benefits also carry over to load and performance testing, where, Jenney says, there is no such thing as a stable environment. “All deployed applications are subject to variable I/O bandwidth, sporadic remote system availability, disk fragmentation, resource corruption.” And performing load tests without taking those things into consideration dramatically reduces the value of those tests. “After all,” he adds, “What good is it being able to support 50,000 users if the system falls over when it gets a bad packet, can't read a file or runs low on memory?”
A true test of an application is one that sees it perform as it would in the real world. “Making servers crash by doing things like fuzzing protocols is a classic attack strategy that may result in the application leaving unencrypted and exploitable data behind,” Jenney says. This is not something that any application should do. Testing with the goal of forcing crashes is simple to perform with an appropriate fault injection tool and is easily combined with other existing test processes.
Jenney provides examples of good fault injection tests:
• Linear Memory Availability Cycling – Exposing the application under test to varying RAM availability, with the change being a fixed increment from maximum to minimum and back again.
• Random Memory Cycling – Varying RAM availability, with the change being a random increment between fixed maximum and minimum values.
• Linear Network Up and Down Bandwidth Availability – Exposing the AUT to parallel bandwidth degradation, with the change being a fixed increment from maximum to minimum and back again; and exposed to cross-over degradation, where the up and down bandwidth are opposite.
• Network Packet Fuzzing – Randomly corrupting network packet I/O to watch for unstable behavior or the lack of unstable behavior. In security testing, often the application that doesn’t fail is the most interesting.
Each of these tests should cause interesting behavior in applications under load, Jenney says. “Starving applications of bandwidth or limiting system memory are great ways to force systems to thrash and uncover all kinds of interesting performance problems, and will likely uncover functional problems, too.”
Get Agile in June
The June issue of Software Test & Performance magazine is available now. In this month’s agile issue, don’t miss Rob Sabourin’s Scrum migration war stories, Glenn Jones telling you how to build a testing process with constant feedback and validation, and much more. Download the PDF now, with no charge and no registration!
Share this link: http://www.sdtimes.com/link/32286