Defect Tracking Lacks Appeal But its importance is at a premium

By Geoff Koch

August 1, 2007 —

Defect tracking may be among the least glamorous aspects of software development. The whole endeavor evokes the image of a stuffy schoolmarm scolding students for their sloppy handwriting, careless mistakes on their homework and stubborn refusal to follow classroom rules.

No doubt some of those kids who decades ago bristled at rote approaches to learning grew up to be proponents of things like Extreme Programming, agile processes and open source. In the technology environment they’re helping to create today—one that increasingly celebrates good-enough, process-light and pretense-free ways of delivering code—defect tracking can seem old-fashioned and, well, maybe even a bit irrelevant.

However, software professionals contacted at several leading providers of defect-tracking-related tools and services brushed aside suggestions that their importance may be flagging. On the contrary, many said that increasing industry complexity is putting a premium on processes of all sorts, including logging and prioritizing issues.

Among the themes that emerged from interviews conducted via a shared Google document that takes up 13 pages and includes more than 7,000 words of responses and riffs on the subject: Defect tracking is less about listing bugs than fostering communication among everyone, from developers to salespeople to customers; defect-tracking tools are indispensable to efforts to reproduce bugs and perform regression testing; and such tools should always be flexible enough to adapt to the developers using them, not the other way around.

Axosoft CEO Hamid Shojaee echoed the views of several of his industry peers that a move to adopt lightweight and flexible coding processes is among the key trends in software development. All processes, including defect tracking, are related to overhead, and it’s controversy-free to agree on the importance of reducing overhead.

“There’s a catch, however,” said Shojaee, founder of the Scottsdale, Ariz.-based provider of bug-tracking and project-management applications. “Every team—depending on the number of individuals working in that team, the complexity of the applications they are developing and the impact of a software failure, such as a Space Shuttle crash if a software failure occurs—has a different ideal point for the amount of process that would make that team most productive.”

For Alberto Savoia, CTO of Mountain View, Calif.-based Agitar Software, it’s most productive to track a laundry list of details about each logged defect—a view that flies in the face of recommendations made by agile zealots.

“When it comes to bugs, I not only believe that they should be tracked, I believe that we should be tracking them more closely and include more detail in bug reports,” said Savoia, whose company sells products that facilitate and augment unit testing.

Recording more extensive information about bugs makes it easier to determine effectiveness of a novel methodology or tool, Savoia continued. For example, a team that implements developer unit testing as a way to identify bugs earlier in the development cycle could use its defect tracking tool to make sure that the end result was fewer post-release complaints and more issues logged during development and quality assurance, or QA.

Nathan Rawlins is yet another bug-tracking pundit who declared process to be more essential than ever. Rawlins’ role gives him sufficient street cred to make that claim. He is a senior director of product marketing at San Mateo, Calif.-based Serena Software, a company that generated more than US$250 million in revenue in fiscal year 2007 selling application life-cycle management software, a category that includes defect tracking.

“It may seem counterintuitive, but as individual development teams adopt ‘just-enough process,’ process actually becomes more important,” said Rawlins.?“The coordination of rapid development processes with other processes, such as testing processes and operational release processes, is often overlooked when implementing defect management. Without effective coordination, the entire development life cycle suffers.”

Get the Bug Bucket
To understand the waxing influence of defect tracking, it is useful to take a brief historical detour. In the 1980s and 1990s, many companies cobbled together their own defect tracking systems. Implementations varied, but the general approach was to create a shareable list of issues caught by testers. Because the intent and goals of the software project were hermetically sealed in the requirements document, the QA team merely engaged in the plug-and-chug exercise of logging where code fell short of requirements.

Needless to say, things are different today.

“The tracking landscape has shifted from disconnected, mostly reactive systems to more integrated, proactive items,” said Paul Underberg, a senior product manager at TechExcel, a vendor of application life-cycle management and service and support management products. “Now that teams are moving toward more lightweight and integrated processes, issues must be tracked within their context: the specifications, requirements, documentation and other collateral that is needed not only to reproduce an issue, but to solve it while still maintaining the concepts behind the intended behavior.

“Issues are also tracked at every phase of the project, and the system is utilized by project management, development and QA in equal amounts,” he added.

Indeed, though it’s still easy to find lists of products for inputting and sorting software bugs, the “defect tracking” label is starting to feel more like a nod to historical tradition than contemporary reality. It’s possible the category is being co-opted, a fate that’s more common, or at least more visible, in the world of mass-market consumer software.

Flickr, acquired by Yahoo in 2005, provides one of the best recent examples of the phenomenon. The popular Web-based photo-sharing and tagging tool was launched in 2004 as a tool created for Ludicorp’s Game Neverending, a massively multiplayer online game. Flickr proved more popular than the game itself, which was soon shelved by the Vancouver, British Columbia-based Ludicorp.

Of course, most defect-tracking tools haven’t shelved their core functionality of allowing developers and testers to input bugs. Still, it’s clear that such tools are evolving to serve a different role, one that’s more about communication than code.

“Hardly anyone uses defect-tracking systems to track only defects,” said Daniel Neades, director at Araxis, based on the U.K.’s Isle of Man. “We actually enter issues into our system for pretty much anything that we need to do in the company. Something needs to be changed on the Web site? Enter an issue. Need to create a marketing document? Enter an issue. End-of-year accounts due soon? Enter an issue.”

This approach, he continued, is simply about prioritizing to-do lists, not forgetting anything important, and giving everyone on the team a reasonable idea when assigned tasks and projects are likely to be completed.

“Oh, and it also tells us how much it cost to fix each issue,” added Neades, whose company sells software for tracking issues, managing projects and comparing and merging files. “It can be quite sobering for a developer to realize that the time taken to fix a bug he caused cost the company $5,000. Developers deserve to have that kind of information.”

As more users from QA to customer support to marketing have begun to find information in defect-tracking systems to be useful, the pressure has mounted to link such systems to a wider array of tools. This is not a trivial task, and vendors find themselves bumping up against another industry truism: While human beings are remarkably adept at wresting useful data from even clunky interfaces, automated machine-to-machine communication is almost always trickier than it sounds. See the yet-to-appear semantic Web as case in point.

“Another trend is the growing integration and even combination of defect-tracking tools and software configuration management systems, such as Subversion and Perforce,” said Neades. “I think everyone in the industry understands the desirability of this. Fundamentally, I want to be able easily to answer the question, ‘What source code changed to resolve this issue?’ We’ve not yet addressed this in our own product, but it is high on the list of things we know we need to do.”

“Defect management is closely tied to many other enterprise disciplines and systems,” said Serena’s Rawlins. “An effective system will automate the connections to other systems and processes and coordinate the collection and handoff of defect information—freeing the project team to focus on creating effective, quality applications.”

One example of Serena’s integration efforts: The company was the first to provide change management tools that directly integrate with the SAP R/3 environment. The tools, under Serena’s ChangeMan brand, allow for scheduling and coordinating of deployment of SAP and non-SAP changes into target environments.

Bug-Logging Norms
Of course, integration is an oft-cited goal throughout the tech industry. Software architects and evangelists invariably promise more plug-and-play functionality than is eventually delivered. The more blustery claims about defect tracking morphing into application life-cycle management software can obscure some of the basic bug-logging techniques that are norms in the industry.

Specifics vary, but every tool allows for issues to be entered and priorities assigned. After encountering a bug, one of the first decisions a developer or tester has to make is just how much detail to include.

“[Being] process-light is always desirable, so I tried to simplify the main defect form, uncluttering it of unnecessary fields,” said David Atkinson, head of testing at Red Gate Software, which uses the JIRA bug-tracking application provided by Atlassian. “My rule is, if you don’t need to report on the field, consider not having it.”

Despite the fact that Red Gate makes much noise on its Web site about its unique approach to relentlessly testing code, Atkinson went on to describe a process that should sound familiar to anyone who’s ever used a defect-tracking tool in earnest.

Bugs are identified and entered by testers and then prioritized and assigned out to developers by a project manager. Atkinson’s testers are encouraged to log even those bugs they know will not be fixed in the current release in a “future” bucket. This obviously takes time and resources but ultimately is more attractive than the alternative: forgetting a bug’s symptoms, the steps to reproduce it and other relevant thoughts and observation of the tester.

At the start of a new release, a project manager is assigned what Atkinson acknowledged to be the unenviable task of reviewing the entire list of “futured” bugs. The Red Gate ethic of recording everything means many of these bugs are decidedly minor, a fact that leads to occasional pressure to close the bugs and make the list more manageable.

“As testers we have resisted this, arguing that if it’s still a valid bug, it should be left open,” said Atkinson. “The compromise was to create a ‘Future—Won’t Fix’ bucket, which is ignored for minor releases but may be occasionally considered for a major release.”

It’s common sense to tackle the high-priority bugs first and leave the rest for later. In fact, some in the agile crowd suggest that in the new everything-is-beta world, medium- and low-priority bugs should be permanently ignored. Atkinson blanched at this idea and instead offered a suggestion for tool vendors for their next-generation products.

Developers, many of whom prefer to tackle medium- and low-priority issues in the same functional area as their assigned high-priority bugs, need a way to edit and annotate bugs in a list that is meaningful to them. The idea is something akin to user-generated tagging that was made popular by services like Flickr and del.icio.us.

Today the only option for developers looking to tackle bugs by functional area rather than priority is to sort bugs using the “component” field. Atkinson noted, however, that “this doesn’t always group together bugs in the same code, and it’s a chore to edit bugs and change this for each one merely to group them. If a tool offered a way to drag and drop bugs in a list, and remembered where they were, internally setting a secondary priority specific to the user, it would be most welcome.”

Where Regression Is Good
Among the many benefits to developers coding in an organization that uses a defect-tracking tool, the ability to easily perform regression testing tops the list. Regression testing aims to catch the inevitable re-emergence of bugs as any codebase evolves. At least in theory, after each new fix or feature is added, the entire batch of previous test cases should be rerun to ensure that no new problem has emerged. Of course, the reality of constrained resources and looming deadlines makes this comprehensive approach impossible. The best most developer teams can do is to make educated guesses, a process that’s made vastly easier by a robust defect-tracking tool.

“If the QA group doesn’t know what changed and the change’s effect, they must retest every feature,” said Richard Riccetti, president and CEO of Seapine. “It is dangerous to take the word of the engineer that the code changes, no matter how small, have no significant impact.”

“The time-consuming thing in bug tracking is not entering the bugs and marking them fixed; it’s the time it takes to create a reproducible test case and verify that the bugs have actually been fixed,” echoed Agitar’s Savoia. “These tasks should not be bypassed or conveniently forgotten, and bug tracking helps you make sure you don’t forget about them.”

Like many a teamwide or organizational tool, a defect-tracking application gets more valuable as more people use it. Attracting a large cohort of users starts well before the tool is purchased and installed. Perhaps paradoxically, the needs of developers shouldn’t always come first as various tools are evaluated.

Easy to Use And Customize
“Development teams shouldn’t evaluate the tool on their own; the extended project team needs to be very involved in the selection process,” said Rawlins.?“Otherwise, the development team may end up with a tool that helps them keep track of their bugs, but still leaves them spending a lot of time coordinating efforts with other project groups.”

No matter who is doing the evaluating, any tool must meet at least two criteria, according to those interviewed. The first, said Axosoft’s Shojaee, is that it should stay out of the way of the user and be easy to use with little or no training.

Strive for “one-click rather than two to get the information you need, two input fields rather than three to record the information, etc.,” he said.

The second must-have is easy customization to a team’s specific needs and preferences. And it’s this point, emphasized repeatedly, that should be enough to make some of those now-grown-up rule-breakers smile. Of course, discipline still matters. But success in software today is less about coloring inside the lines than carving out an independent and often highly specific niche.

“The days of companies changing their processes to support tools are nearing an end,” said Riccetti. “Competitive advantage is derived from doing things differently from everyone else. The ability to define and change your process helps deliver that advantage.”

In other words, the my-way-or-the-highway misanthropes are getting the last laugh on their super-strict schoolmarms, after all.

Share this link: http://www.sdtimes.com/link/31026

News Briefs: October 15, 2008 Electric Cloud makes its product line compatible with Microsoft's HPC Server, RapidMind released add-ons for its RapidMind platform, and Fujitsu opens a subsidiary in North America.

Integration Watch: Defect tracking springs to life Instead of building your own defect tracking-tracking systems, look at the growing number of options already out there

BMC Software reworks AppSight for mixed environments AppSight's workflow automation module now supports Java EE applications, making it possible for AppSight users to test application behavior across platforms without using separate products.

Add comment

Name*
Email*
Country

Comment
Preview

News on Monday
more>>
SharePoint Tech Report
more>>

Download Current Issue
ISSUE 3/15/2010 PDF

Need Back Issues?
DOWNLOAD HERE

Receive the print Edition?

Google Code turns 5
Google Code Turns 5, and adds a Paxos Algorithm to make the system more stable and reliable.
03/17/2010 11:16 AM EST

Test your Visual Studio 2010 know-how
Microsoft is offering free beta certification exams for Visual Studio 2010.
03/17/2010 11:08 AM EST

Microsoft lifts the hood on IE9
Microsoft is previewing IE9.
03/16/2010 01:10 PM EST