LOGIN | REGISTER NOW | SUBSCRIBE
 
 
News on Monday
more>>
SharePoint Tech Report
more>>


   

 
 
Download Current Issue
ISSUE 2/1/2010 PDF

Need Back Issues?
DOWNLOAD HERE

Receive the print Edition?


 
blogs tab
Visual Studio 2010 Release Candidate Available Today
A Visual Studio 2010 release candidate is available on MSDN.
02/09/2010 09:45 AM EST

Is Microsoft eyeing Office subscription pricing?
Microsoft may be preparing to offer a new Office pricing option called "union," which charges the same for cloud as on-premises.
02/01/2010 09:38 AM EST

Facebook rewrites PHP runtime
Facebook is about to open source its own PHP runtime, written from scratch for speed.
01/30/2010 08:53 PM EST

 

Events calendar tab
Macworld 2010
2/9/2010 to 2/13/2010
San Francisco
IDG World Expo

SPTechCon 2010
2/10/2010 to 2/12/2010
San Francisco
BZ Media

PyCon 2010 (Python Conf.)
2/17/2010 to 2/25/2010
Atlanta
Python Software Foundation

Southern Calif. Linux Expo
2/19/2010 to 2/20/2010
Los Angeles
SCALE

IBM Pulse
2/21/2010 to 2/24/2010
Las Vegas
IBM


 
print Printable version 
Most Read Latest News Blog Resources
Facebook brings HipHop to PHP
HipHop compiles PHP to speed up the Web layer at world's most popular social networking si...

Adobe moves LiveCycle application platform to the cloud
Release gives developers just what they need to create targeted business process-oriented ...

Oracle acquires AmberPoint
Today's merger will probably affect most of AmberPoint's existing partnerships with other ...

Digg!  Digg
Reddit  Reddit
Slashdot  Slashdot
Share on facebook  Facebook
Share on Twitter  Twitter

Hitting the right notes in QA


By Jeff Feinman
August 15, 2009 — 
If the software development process weren’t buoyed by testing and quality assurance metrics, there would be a great deal of guesswork taking place.

It would be like a symphony orchestra getting up on a stage without a single sheet of music and trying to play Bach or Beethoven.

Only with software, the symphony is getting more complex each year. With a constantly evolving software development process that includes different IDEs, systems integrations and a seemingly endless stream of new features, it can be difficult to maintain a sense of the values behind metrics.

According to multiple testing and quality assurance software providers, good metrics should provide adequate information on a defect’s effect on the overall costs of producing the software or the likelihood of the application failing. Good metrics should also offer an idea of how compliant a developer is with industry standards or how secure a developer’s code is.

Bill Curtis, senior vice president and chief scientist for application management company Cast Software, agreed that one of the most important metrics of an application’s health in the long term is the percentage of defects detected before a developer gets to test.

“That’s the best prediction of long-term improvement in the quality of your software. Defects cost much less to fix in the design and coding phase than in testing, and the earlier defects are found, the cheaper they are to fix,” he said.

Chris Wysopal, cofounder and CTO of Veracode, said that while things like authorization problems can certainly be tested for in the final build or when an application is deployed, the cost to fix those problems is very expensive.

“The best time to look for authorization problems is at design time,” Wysopal said. “At that time, you can do threat modeling, which is inspection more than testing. Then you have defects that show up when you’re writing the code, like buffer overflow, and that could be found by static analysis at code design time.”

Threat modeling is the process of assessing and describing what attacks a piece of software is vulnerable to in order to eradicate potential threats.

Key metrics
As developers test as early as possible in the application life cycle, what are the main metrics around software quality that they should be aware of? Should they be eying statement coverage? Cost per defect?

Code coverage, a measure that describes the degree to which an application’s source code has been tested, is a time-honored metric that still holds a lot of weight among QA professionals. However, it can paint an incomplete picture in connected systems, an expert says.

“Code coverage is meant to give a comfort level that you’ve exercised the various bits of code,” said Mark Sarbiewski, Hewlett-Packard’s senior director of products. “It’s an interesting metric from the angle of security. Code coverage tells you something about the attack surface of an application. It might tell you about backdoors and weaknesses that wouldn’t be exposed if you treated it like a black box.”

Code coverage, though, can be an incomplete metric because of the many approaches that developers can take to build software programs, he said. In some cases, developers compose applications where they are utilizing services without having access to any source code, and it has become common for software to be integrated with other systems, he added. Developers need to know about the code in order to do proper code coverage, he said.

Brian Chess, founder and chief scientist of Fortify Software, agreed that code coverage has value in security scenarios, and he said code coverage is good for directing testing. Statement coverage, which is part of the code coverage criteria that asks if each line of source code has been tested, is “really good for telling you things, like when your tests haven’t even turned on half of the application yet,” he said.

Naturally, another important quality measurement is cost per defect, as it involves an organization’s bottom line. Chess said he has commonly heard stories for more than a decade about the cost to fix defects when a developer is coding versus after they’ve released to production.

“When you try to do that yourself, you find out it’s difficult to do a controlled experiment where you have a large enough sample of defects," Chess said. “And people have kept track of how much time and money they spent fixing those defects to reproduce those.”

Veracode’s Wysopal said the impact of cost per defect is based on the type of defect found and the point in the life cycle where it’s found. Certain defects, like authorization problems, are found late in the process and are very expensive, as a developer has to rethink the design of his or her authorization scheme. But something like integer overflow, on the other hand, can be very easy to fix, because the solution merely involves inserting one or two lines of code.

Finding the origin of defects
A defect may be more than it seems. Oftentimes, defects are a window into chronic development problems.

When it comes to metrics, Fortify’s Chess said that it is always important to understand where bugs and failures are coming from and what sort of bugs developers are writing.

For instance, if buffer overflow is the mistake people are making over and over again, a manager might want to consider moving to a set of libraries that can eliminate that problem. “I think there are really some interesting things you can do by keeping track of the kinds of defects you’re finding in your code,” Chess said.

The origin of defects could also trace back to design time. Mark Eshelby, director of product management for Micro Focus, which acquired Compuware’s quality software line in May, said defects introduced in requirements and found by end users cost the most to fix. At the same time, a requirement defect found and corrected in requirements costs the least to fix.

“Defects are bad, but defects in critical business requirements or goals are really bad,” Eshelby said. “The mainstay metrics of any testing and development organization are typically defect metrics. For many organizations, this is their only metric, but defect metrics by themselves do not provide an accurate assessment of application quality. It's important to look at the defect metrics from the perspective of the [business] questions you are trying to answer.”

Opposite from requirements in the life cycle is mean time to failure, which looks at the speed and frequency of failures after an organization decides to run an application. Mean time to failure takes into account under what circumstances a program fails and where in the SDL there failure took place.

“Those are variants of the post-release issues that we think are hugely important metrics to understand,” HP's Sarbiewski said. “Connected with that is finding out the nature of the defect. Is it requirements, design or strictly a coding defect?”

Sarbiewski said mean time to failure works on two angles for HP internally, both on a functional level and performance level. Mean time to failure can show if something isn’t functioning correctly and causes failure, and if an application crashes due to load.

In addition to metrics such as mean time to failure and cost per defect, a good way to mark improvements in software quality, according to Cast’s Curtis, is by looking at defect density, or defects per function point. “You can benchmark against it,” he said. But it is not a foolproof metric.

The potential problem with that is when someone is comparing against test cases, they’re limited by what the people who wrote the test cases knew about the system requirements, Curtis added. If there are things the test case writers didn’t understand within the requirements, a tester won’t get the right test cases and will have problems within the code.

“The best measure ultimately is looking at only defect density and operation,” he said. “'Given the first six months of operation, what did we see?’ That’s getting a good idea of how effective we were at removing defects before we went into the field.

"I’d want to see well over 90% of the defects captured before I get into test, and I’d want to see 98–99% or higher captured before we go into operation."

Indeed, today's metrics have made it possible for developers to eradicate more defects before code is shipped to the end user.

Maintaining quality metrics means quality code
There’s been a lot of progress in quality metrics, said Curtis. He also said he sees improvements in measuring the structural aspects of code to determine if it has flaws that could lead to problems later on.

With that, Cast focuses on determining if an application is high quality so that an organization’s cost of ownership remains low.

“[Quality metrics] are a whole new set of metrics different from test coverage or defects per line of code that really look at the nonfunctional aspects of the system: the structural aspects, the engineering aspects, the architectural aspects,” Curtis said. "And [they] begin to evaluate whether this is a good system or one that’s teetering on its last legs.”

David Bressler, a product evangelist with Progress Software, said an important thing to keep in mind while testing is the ability to seamlessly fix a problem and quickly create a policy around it so it doesn’t happen again. This will help improve the quality of future development, he said.

Chris Kraus, a product manager with iTKO, meanwhile, said that people are getting more progressive in the way they’re thinking about the quality of an application, because the way they’re building applications is changing.

“With multi-tiered, multi-layer, integrated environments, QA is now measuring multiple things,” he said. “They’re now looking at making sure systems are integrating properly and business logic is working properly, so the number of things they’re measuring is increasing because quality is having to adapt to changes in application writing.”

Some of the things developers are checking with more complex applications and environments include system integration tests and user acceptance tests, which involve a product owner or client giving approval to a modification or addition to the system, Kraus added.

Quality can also mean compliance with best practices and guidelines, which change how efficiency is measured.

Metrics for measuring compliance
Bressler said that the main metrics in his eyes are the "number of rules per policy" or "per standard." Metrics like those give an idea of how compliant a developer is with coding practices from organizations like the WSI (Web Services Interoperability) organization, which promotes Web service interoperability across platforms, operating systems and programming languages.

“That could show up as the number of errors or a percentage of the overall spec, but there are metrics that are a bit harder to measure in terms of developer productivity,” Bressler said. “You have metrics in two different dimensions: metrics that focus on how compliant developers are, and metrics on how successful they are as a software developer.”

Compliance doesn’t always align with a developer’s work to an organization's guidelines, like the WSI. ICSA Labs, an independent division of Verizon Business, sets guidelines for commercial security products and specializes in research and certification testing for security. ICSA Labs works with product developers to create a set of standards and requirements for security.

“Those cover everything from documentation on the product to something as simple as the completeness of that documentation,” said George Japak, managing director of ICSA Labs. “It’s amazing as to how there’s a disconnect in the documentation that developers pool together, and it’s not necessarily in sync when the product goes to market.”

ICSA Labs reads a product’s documentation for completeness before the documentation and requirements are used to configure the product. Japak said that in many cases, the documentation doesn’t map correctly to the product and the functionality, or it’ll even reference functionality in the product that doesn’t even exist anymore.

What metrics mean to your organization
In today’s software development process, there are two ways to look at application metrics, according to Veracode’s Wysopal. A development team can look at it from an organizational standpoint, understanding how efficiently an organization is working, or on an individual basis, drilling down to one piece of software and figuring out if there are too many defects in it.

For example, Wysopal said that Veracode applies the Common Weakness Enumeration (CWE) to each defect found. The CWE is a community-developed list of software weakness types maintained by nonprofit technology organization MITRE to provide a common standard for describing security weaknesses in architecture and code.

“If we find an integer overflow and label it with the CWE, right away I can track different classes and count how many defects I have in a particular class,” Wysopal said. “But organizationally, I can see what [defects] are skewed higher than they should be. I can then compare that to the industry average.”

A valuable metric, whether it is code coverage or defects per function point, is key to mastering a perfect symphony, free of slips and errors. That way, the entire process can remain in tune.


Related Search Term(s): QA


Share this link: http://www.sdtimes.com/link/33688
 

Add comment


Name*
Email*  
Country     


  • Comment
  • Preview
Loading