Stalking The Wild Software Defect

By Esther Schindler

August 15, 2002 —

Some development managers release impressive software armed with only a spreadsheet, a pile of Post-it notes and a confident attitude. Others, equipped with every QA tool imaginable, have never met a deadline. Clearly, bug-stomping success isn't the result of tool quality, but of your ability to get the most out of the tools available.

According to successful bug hunters, the key to effectively using QA tools (by which we mean anything in the configuration management pantheon, from bug-tracking databases to version management to source-code control) is to view the system as something more than software. "Like e-mail, defect-tracking systems are inherently a vehicle for carrying on a conversation that's disjointed in location and time," explained Elton Hay, a consultant who specializes in QA tools. "[The] ability to facilitate is one of [their] main functions." Ideally, QA tools enable team members to communicate about a project's status, control the process a bug goes through, avoid redundant work, and engage the development staff in taking ownership of the code.

Bug-tracking applications control what happens to any bug entered into the system. That process, at best, can reflect only the efficiency of your organization's workflow. "The software enables us to implement the QA process we have in place," said Mike Cooper, QA manager at financial consulting company Revenue Technologies Corp., which uses Seapine Software Inc.'s TestTrack. "You have to know that before you can do anything else."

So, the first step is to determine that workflow. "What is the current development process? Is it documented? Is it followed? Is it followed consistently throughout the entire organization?" asked Mark Griffith, senior consultant at Merant International Ltd. The more "yes" answers provided, the easier it will be to deploy and use the chosen QA tools. "Draw a physical flowchart or workflow diagram," advised Cooper.

As part of the workflow process, consider which staff members need to see which items, who supervises the process, who assigns bugs to be fixed. "Defining the flow of work through the software is probably the most important part of using it," said Jay Varner, director of programming at Data Information Management Systems, which makes election management software and relies on Red Gate Software Ltd.'s tools.

When defining workflow, take a wide view. That's the lesson learned by Pam Pullem, director of IT projects at mortgage insurance firm PMI Group Inc., when the initial scope of the company's software expanded. The QA team had implemented its Sesame ExtraView application to reflect its own hierarchies, but those categories and workflow settings didn't serve the needs of the production support department. While ExtraView permits Pullem to make the changes necessary, she said she wishes she'd started out with a broader perspective.

Bug-tracking tools can do two things, Griffith said. They can implement, enforce and control your current development process (and that alone should improve your time-to-market). Plus, the software can be part of a drive toward process improvement. But trying to do both at once can cause huge growing pains, developer resistance to change, and confusion about the team's goals.

A manager may find that changes to the company's process are necessary to use the tool. That could improve the development team's ability to deliver, but wholesale, across-the-board changes to process should be avoided.

On the other hand, managers need to be ready to take advantage of the QA tools when changes to the operation are required. Adam Kolawa, CEO of code management tool vendor Parasoft Inc., has a firm policy that the software enforces: You can't check in code that won't compile. Jim Beveridge, president of software vendor Connected Software Inc., has two main rules for ensuring code quality: First, "programmers fix their own bugs. Period. This makes sure that your good people aren't penalized for being good by being forced to fix other people's bugs." And second, "programmers don't get to move on to a new feature until QA says they are done with the previous feature. This makes sure that your poor programmers don't constantly have the pleasure of working on new features while everyone around them fixes their bugs." These two steps, he said, force a continual QA process instead of a "test it at the end" mentality.

DON'T GET CARRIED AWAY

Start small. Don't think about integrating with other tools initially. Keep it simple. QA managers said repeatedly to get the basics working smoothly before tackling complexities. And even then, they said, stay conservative.

"Just because our software lets you create customer fields doesn't mean you have to use them," advised Neil Davidson, technical director at Red Gate. The greater the number of arbitrary fields, he said, the more likely users will be to leave them blank, or to enter junk data. Not only is wrong nformation worse than none, but the increased complexity becomes a barrier to the developers and testers who need to use the system-and that's what this whole exercise is about, he said. "Designers readily adopt and demand tools which improve their productivity," said Art Pina, a longtime software developer and CAD engineer. "If we perceive the gain to the project will outweigh our personal inconvenience, we'll accept it."

Take time for training. While budgets are tight, and QA tools typically get less respect (and thus money) than they deserve, realize that the tool's value will reflect the common understanding of its users. Revenue Technologies' Cooper has quarterly review sessions in which he sets guidelines for, say, what constitutes a high-priority bug.

Conduct a design review for the bug-tracking system, suggested Jesse Keller, senior software engineer at ISE Research, to ensure that the system reflects reality without overburdening users with irrelevant data. That sanity check can teach you plenty of simple usability lessons. One example: Show items in the order they'll be encountered, rather than alphabetical.

Merant's Griffith recommended that the company take a few days of consultancy, so an expert can challenge your assumptions and ask the right questions. That's important, agreed Hay, because the systems' users need to establish a common vocabulary in dealing with the bug-tracking process. One large multidivisional client, Hay said, spent a year-and-a-half arguing over the bug-tracking definition of the word "closed." Eventually, they agreed that it meant "the company won't spend money to fix it."

Assign issues to the correct people, based on their expertise and responsibility. If your software enables you to automatically assign tasks based on the text of the bug report, such as "anything that mentions performance should be routed to Laurie," take advantage of that feature. Because programmers like to help their friends, said Cooper, a developer is apt to take on bug fixes even if he isn't the appropriate person. Whether it's accomplished in software or more usually by a project lead or QA specialist, it's time-effective to establish a firm assignment process.

TRACKING TIPS
If you do a good job of categorizing bugs, you'll get more than a set of pretty management reports, useful metrics and a sense of satisfaction from contemplating what your team's accomplished in the past few weeks. Effective tracking and priority-setting lets developers know which of the outstanding issues will be fixed in the next release, said Data Information's Varner, and it helps them set deadlines.

But be careful about what goes into the tracking system. Don't put in bugs that will never be fixed, said Red Gate's Davidson; they just clutter up the system and lower morale. Also, decide when a bug ought to be entered into the bug-tracking software. Many developers tend to use the software to record bugs in their own code as a sort of to-do list, said Steve Schimeall, senior consultant for Segue Software Inc. These bugs, which the developer will fix herself and nobody else will see, are of questionable value to the organization because they don't impact anybody else. Plus, Schimeall said, "honest guys tend to lose by doing this," because they're confessing minor errors to the whole team. Instead, he suggested, enter bugs only when the integration affects more than one person.

However, don't leave out suggestions and feature requests. New ideas should be entered in the system, even if they're immediately marked as "deferred to the next version," said several managers. It can be useful and sometimes important, though, to give such suggestions a time-sensitive deadline, or these "wishes" will fall to the bottom of the priority list and stay there.

Don't underestimate the value of peer pressure. Anybody who's working on a project should be able to see the reported bugs, managers said repeatedly, both for general communication purposes and for the development staff to police itself. "Everyone picks on the bug generators about 'another round of bugs created by so and so,'" said Connected Software's Beveridge.

It's important to establish which data will be visible to which users. For example, customers who enter bug reports over the Web shouldn't see a bug's priority setting. You also need to define who is expected to set those priorities and who is permitted to close bugs. According to Geoff Schardein, senior director of sales at Seapine Software Inc., developers should never be allowed to close their own bugs. "People don't catch their own mistakes," he said.

Ensure that people aren't overwhelmed by information irrelevant to their jobs. Keep people informed, but don't overload them. At PMI Group, Pullem said, developers see only the bugs related to their own projects. If the system can send out e-mail notification of status changes, send them only to the developers directly affected, and to the project and QA leads. Otherwise, suggested Schardein, everybody will be deluged with e-mail and won't read any of it.

Overall, stay aware of what your QA tools can do, and what they cannot. "A bug-tracking system is like oil in an engine," said Beveridge. "Oil allows an engine to operate fast and efficiently, but oil can't fix a damaged engine, can't make a lawnmower engine into a Ferrari engine, and most definitely can't keep the driver from running into a wall."

Esther Schindler is a freelance technology journalist.

Share this link: http://www.sdtimes.com/link/26693

SOA Software lets developers check out candidate services Portfolio Manager visualizes interrelationships between business imperatives and candidate services to help in planning

Software AG’s IDE evolution is only Natural NaturalONE aims to preserve Natural's install base by appealing to Java developers working with existing applications

SOA Software gets into cloud service governance The company's new suite of Atmosphere products will provide management, monitoring and provisioning of cloud infrastructures

Add comment

Name*
Email*
Country

Comment
Preview

News on Monday
more>>
SharePoint Tech Report
more>>

Download Current Issue
ISSUE 3/15/2010 PDF

Need Back Issues?
DOWNLOAD HERE

Receive the print Edition?

Google Code turns 5
Google Code Turns 5, and adds a Paxos Algorithm to make the system more stable and reliable.
03/17/2010 11:16 AM EST

Test your Visual Studio 2010 know-how
Microsoft is offering free beta certification exams for Visual Studio 2010.
03/17/2010 11:08 AM EST

Microsoft lifts the hood on IE9
Microsoft is previewing IE9.
03/16/2010 01:10 PM EST