What are the best practices for automated Java testing?
Stories Columns Opinions Resources
Preflight builds spread wings for smoother projects
Developers are increasingly turning to preflight builds, allowing them to experiment with ...
|
Coverity creates program to enforce code adherence
The Architecture Analyzer uses mapping technology from the company's Software DNA static a...
|
QCon 2008 features domain-driven development
This year's QCon invites speakers like Eric Evans and Dan North to talk about domain-drive...
|
.NET similarities prove golden for Silverlight
Microsoft has focused on making Silverlight 2 symmetric with the .NET platform, and that h...
|
SOA Watch: New economic realities
In the current economic downturn, agile programming and SOA are attractive options that bu...
|
Integration Watch: A new twist on threads
The key to raising the efficiency of multiprocessors is to shrink the overall workload by ...
|
Integration Watch: The Return of NetRexx?
Java scripting languages are seeing a surge in popularity, with NetRexx looking particular...
|
Windows & .NET Watch: Transaction crowd gets a boost
With multicore chips becoming the standard for processors, the need for a flexible, usable...
|
From the Editors: Election should shake up JCP
Rod Johnson has the right ideas for opening up the Java Community Process, and he may be a...
|
Letters to the Editor: Sun gives REST, SOAP choice
A reader takes issue with a headline on our story about Sun working with REST along with S...
|
Guest View: Be smart and lazy
The optimal solution for problems is the simplest one, so always aim to streamline your ap...
|
Zeichick's Take: From EXEC to EXEC 2 to REXX to NetRexx
Andrew Binstock's column last week, "The Return of NetRexx," brought back some fond memori...
|
Advanced Corda CenterView™ Data Visualization for the BusinessObjects™ Intelligence Platform
Corda Technologies presents a white paper on pervasive BI. The BusinessObjects business in...
|
From Mobile to SOA: A Guide for Optimized Application Deployment
Customer need has driven the emergence of multiple computing tiers. Today’s application d...
|
e-Kit: Web Application Security
Is your network secure? What about your web applications.
If IT security is your top p...
|
Practical tips for saving money on code maintenance
If software design is expensive, well, code maintenance is even more so. When you look...
|
By Lisa L. Morgan
April 1, 2008 —
If automated Java testing is so cool, why isn’t everyone doing it all the time? It boils down to comparing the perceived value of the up-front investment in test automation against the long-term productivity benefits. Too many organizations focus on the investment, training and learning curves, rather than on how that can be leveraged for months, or even years, to come.
“A lot of developers and testers learn bad habits at an early age,” said Frank Cohen, CEO and founder of PushToTest, an open-source Java automation company, and author of “Java Testing and Design: From Unit Testing to Automated Web Tests.” He added, “In Web services, AJAX and SOA, test code is a valuable corporate asset that needs to be maintained.”
Sometimes when developers write short code or think their code won’t change over time, they consider automated testing overkill. According to Alberto Savoia, CTO of automated unit-test company Agitar, automated tests can take10 times as long to set up as their manual counterparts. So if code is static or will be tested only a couple of times, automated testing may not make sense. Problems with that type of thinking can arise particularly in the case of Web services and SOA, where software reusability is central to the development philosophy.
A developer may think that automated testing isn’t necessary, but time may disprove that. The code likely will have to be maintained, and, depending on what kind of software it is, it may become subject to upgrades, patches and service packs. Developers or testers may simply forget that even a single line of code or test code needs to be maintained.
“Automated testing is like putting money in a retirement account,” said Matt Love, a software development manager at Parasoft, which sells cross-platform test automation tools. “It hurts up front but pays down the line if the code stays around.”
Automated testing is virtually a given for large companies with complex code. Because they have teams of developers working on various projects in parallel, it’s only logical to automate testing to achieve better test coverage and to simplify processes over the long haul. The high price of commercial automated testing tools once slowed adoption in smaller development shops, but the broad availability of open-source solutions has lowered the cost of entry. In fact, some tools are available free.
“Nowadays, there really are few cases where automated testing at some level is not a good idea,” said Ken Krupa, CEO and founder of eMethods, a consulting company. “Agile development principles have brought a ‘test first’ mentality to mainstream thinking and, as a result, the software development community has benefited from an explosion of free, open and proven testing technologies created as a byproduct.”
Knowing What to Automate
Regardless of whether commercial or open-source solutions are used, automated testing and manual testing tend to go hand in hand for a couple of reasons. First, it’s virtually impossible to achieve 100 percent coverage manually or in automated fashion, tool vendors said. Second, it doesn’t make sense to automate all tests, because they take considerable time to set up and run. A more effective approach, in terms of technology, time and costs, would be to identify areas where errors and vulnerabilities reside and apply automated tests to those areas.
“You can automate yourself into a corner if you don’t know what to test,” said Dan Gouveia, a Rational Software sales engineer at IBM. “You need to understand requirements, the testing process and the time it will take.”
Nearly everyone interviewed agreed that manual testing isn’t dead—but it’s getting more difficult to justify. The sheer complexity of software and the environment in which it runs makes some types of manual testing nearly impossible. And it does make sense for some functional tests or cases in which human interpretation is critical to user experience.
According to PushToTest’s Cohen, the two critical types of Java tests are functional and performance. The former, including acceptance, regression and integration tests, ensures that the software operates correctly. The latter identifies the metric that lets users forecast hardware, bandwidth, and CPU resources. And scalability issues as they relate to user requirements also are established.
“You may be able to do functional tests manually but you [may not have enough manpower] to adequately cover load and performance testing,” said Cohen. “If your company wishes to succeed in a world where services talk to each other, [you need to understand that the scope and degree of testing] will outgrow manual testers.”
Jonathan Lindo, CEO and founder of debugger maker Replay Solutions, agrees. SOA and virtualization are resulting in new levels of complexity that are too difficult to tackle with manual testing. Some of his customers have nine- and 10-tier applications running on different platforms, creating a matrix of systems. In those circumstances, automated testing enables huge combinations of automatic regression tests.
“When you have applications running rich Java scripts and Ajax, you need to bring up the back end and multiple tiers,” said Lindo. “Once you’ve found the problem, the question becomes how you communicate that problem.”
Moreover, the complexity of databases, Linux, Windows and browsers requires an entire infrastructure to re-create and solve an issue, he said.
Raffi Margaliot, director of products at HP, said that such complexity drives the need for automated test solutions and that the maintenance of automated tests could also be complex.
“There is a need for software test reuse,” he said. “If you structure tests in a componentized way, you can mix and match tests based on business processes.”
When it comes to automation, focus is critical because there will never enough time or resources to test everything, Margaliot said. Instead, companies need to take a risk-based approach to automation, in which tests are optimized for the riskier areas, including those that undergo a lot of change.
Having said all that, there are some circumstances in which automated testing doesn’t make sense:
• If the code is static.
• An ROI analysis reveals automatic testing wouldn’t pay off in the circumstance.
• If the software or software development company is so new that the architecture and interfaces haven’t been defined yet.
In the first two circumstances, time may tell another story. In the third case, it makes no sense to automate tests when the basic foundation for the code itself hasn’t been defined.
One Big Happy Family?
In recent years, the rigid wall between developers and testers has been softening, but it’s not altogether gone. Traditional after-the-fact testing—whereby developers wrote code and tossed it over the fence to testers—is passé, particularly in light of agile development and extreme programming. “Test early and often” is the mantra of life-cycle testing solution vendors and companies that wish to avoid the high costs of bugs that slipped through development.
Parasoft’s Love said that a couple of problems can happen after overnight batch processing: Developers may not take responsibility for the bugs in their code, assuming that someone else will take care of them. Or, perhaps they just “glaze over” at the high bug count.
“If you’ve got 17 failures and no one takes action, before you know it you’ve got 100 failures,” he said. “If you can assign bugs to individual developers and send the report of those bugs to the developer and his manager, then there is incentive to fix the bug or reassign it.”
And who owns bugs after all? In a life-cycle situation, you still have two skill sets: developers and testers. Historically, developers have not been testers, and testers have not been developers. PushToTest’s Cohen said that, over time, more testers are claiming that they’re learning how to code, which is a good thing. Automated testing requires solid coding skills.
Agitar’s Savoia said that a historical program lingers. Many companies underestimate the importance of testing, as evidenced by the “second-class” testers they hire who are paid significantly less than developers. The lesser-skilled testers lack the development skills they need to write adequate tests. High-quality testing and development go hand in hand, which is why Savoia has been known to hire PhDs in critical and highly competitive situations.
PushToTest’s Cohen believes that at least part of the problem is a disparity of resources.
“Testers don’t get all the good stuff developers get,” he said. “Developers get IDEs and conferences like Java One where they can talk.”
Others argue that testers have their own publications and conferences, albeit smaller ones, and that some of the career testers do have a strong sense of community. Those who argue against a disparity of resources acknowledge that test engineers are generally not as highly regarded as developers. Also, because test engineers get paid less, there is greater turnover. The turnover, of course, is yet another argument for automated testing, because the intellectual property resides in the system rather than in the head of an individual who wrote a manual test.
Regardless, testing is no longer the sole domain of the QA department; it is the responsibility of developers as well. Parasoft’s Love says that with Junit “in your face all the time in Eclipse,” it’s difficult for developers to escape it and easy for them to run unit tests. More important, developers are the best qualified to write unit tests for their code because they understand what the code is supposed to do.
At Sun, life-cycle testing is just part of the culture, according to Jerry Driscoll, director of Java SE Quality. His company uses a mix of automated and manual testing, and he said that it’s a lot more efficient to design in automated testing at the beginning rather than try to do manual testing throughout product development and to automate testing after a product has shipped.
Driscoll is a big believer in test automation, given its relatively low cost compared with manual testing, although scalability is also an issue. Since Java is “write once, run anywhere,” Sun needs to be able to maintain software for several versions of Linux, Windows, its own Solaris and more. Sun has processes in place to ensure that software engineers and the quality team work together throughout the life cycle so that “quality is engineered” into products.
Test Automation Best Practices
What are the best practices? The answers vary, but the most common are:
• Whatever the large consulting firms say they are.
• Whatever vendors say they are.
• Whatever other enterprises are doing.
Practically speaking, how is it possible to know what actually is a best practice?
Best practices are generally thought of as methodologies that produce predictable and repeatable results. They can be specific to an industry, company or. They also can be high-level business practices or low-level technical best practices.
“Best practices are like opinions,” said IBM Rational’s Gouveia. “None fit perfectly.”
Through discussion and collaboration, developers have created “patterns” that identify standard ways to achieve some stated goal. For example, a pattern might identify a standard way for a browser to communicate to a server, or a pattern might exist that defines the best way to test an application. In addition to patterns, there are anti-patterns that prove what doesn’t work. Testers need to band together to create their own standards or best practices, just as developers do, PushToTest’s Cohen said.
eMethod’s Krupa also likes patterns because the masses reject and accept what is best based on experience. As the patterns become accepted, they are given names and added to catalog. The catalog of pattern descriptions eventually manifests itself as real code and other downloadable items that can be used. Patterns are constantly subject to change and revision, Krupa noted.
PushToTest has developed its own patterns that enable the company to transform functional tests into performance tests and, later, into service tests. It’s test reusability taken to an extreme. Here’s how it works: The functional test verifies whether the system is operating correctly. By running 10 threads concurrently, load and performance are tested. By running the concurrent tests over time, it is possible to monitor business services. Thus, the functional test becomes a performance test, which in turn becomes a quality-of-service test.
It should be noted that not everyone interviewed for this article identified with the terms “patterns” and “anti-patterns.” But those who are familiar with them know them well. For general best practices, see “10 Java Automated Testing Best Practices”.
As for testing, there’s no standard reference application for Web services, Ajax or Web applications, so there’s no easy way to compare the performance of platforms or identify the scale of an application running on that platform, interviewees say. However, you won’t hear anyone interviewed for this article stating that there are no automated testing best practices. They exist, but there’s no one-size-fits-all best practice.
The types of tests can vary by server, client, and browser-based code, and they really have to do with what’s being processed. On the server side, functional and performance tests are important because the server functionality has to be validated, and the servers have to handle high volumes of HTTP traffic, code running on application servers, and support thick-client applications. On the client side, functional tests are important.
IBM Rational’s Gouveia said browser-based testing is evolving as browsers do. They used to display static content but have become more intelligent with the addition of servlets and Java Beans. In the meantime, Ajax has moved more processing power to the client side and browsers.
“If your organization applied a practice successfully during a project, it should be communicated as a best practice,” said Brandon Jones, senior application engineer and senior consultant at Telelogic. “As other projects apply the practice, it should be improved upon continually. That way, an organization can capture successful approaches [while encouraging] collaboration and agreement about what works and when it doesn’t.”
Related Search Term(s): Java
Share this link: http://www.sdtimes.com/link/31873