09/05/2009 06:36:33 AM EST

IMO the main stumbling block to geting good working software out the door is that the requirements are not well defined or keep changing. That problem will always remain, no matter how far up the lifecycle you move any other activity, and that is what causes so much inefficiency in software development. Let's take the case of embedded system development, where hardware and software are being developed in parallel. The hardware/software partitioning is where the pain is here, and once that is done, the test team goes ahead and makes a model of the software to derive the test plans and test cases, well ahead of the actual target software. Even if the software requirements change, the test team, not constrained by memory or other run time requirements/limitiations, can tweak their model to come up with the new test cases quite easily. However, the long lead times in most nontrivial embedded systems almost guarantee that new functionality is forced into the software and the resulting timing and memory interactions are what make these systems hard to certify, and the model can help only so far in solving problems in these areas. I personally have always used a model to derive and validate test cases. This clarifies my uderstanding of the requirements, and I am free to use any platform to create this model whereas the development team must make their software run on the target platform, which may or may not be availble, may have constraints that I need not apply to my model (immediately). The closer the model is to the target, of course, the better will be the test. Automation is only a step in this process, and can remove the tedium from the testing process. Automation is not a panacea. I wonder how many remember the HOS, later espoused by none less than James Martin; but automatic programming is also only as good as the requirements, no better. Component factories were once thought to be the answer, but while ORBs have made it easier to get a working system together, the finer points of response time, and performance still have to be addressed.

IndiaMallika Chellappa

09/06/2009 10:43:57 AM EST

Mr. Duffe, enjoyed your article on the future of testing. My comment is though that test development in many shops especially when a QA area exists (not very often in my experience) is already performed in parallel with development but maybe not completely like the future you envision. It is typical say in our shop to document the test cases before development has even begun, not necessarily the future but these are visible to the developer and helps to improve productivity and eliminate some of the common looping. In my opinion even if it were possible I am not sure it would be very productive to try and develop tests before some semblance of an application exists, I can’t speak to service or device development but for a typical say windows application as you mention a test tool will not work as there is nothing to target. I definitely agree with where I think you were going that to eliminate the traditional looping in the QA process would save time and improve the efficiency and quality of development but the reality is different. Regardless of the development methodology, changes to specs and requirements will happen, looking at our own application development even if we could automate our tests scripts before the application is complete we would not because as the final application evolves I would be expending effort in modifying the QA tests which is of little value and would eliminate any potential cost savings. There are ways already being practiced that do allow the achievement of some of the results that you mentioned some such as unit test automation put the power back in the developers hands but still haven’t seen wide spread use. Thanks again for the good article definitely thoughts and ideas to consider and make the development process more efficient.

United StatesPhillip Avelar

09/14/2009 09:44:07 PM EST

An interesting and thought provoking article. I think most readers familiar with QA/automation would, like commenter Phillip and myself, would say that it would seem quite impossible to automate before the product or features are even available, unless there is a breakthrough in test tool capabilities in the future. However, I do agree that bringing the QA/testing process up front with development cycle rather than at the end will greatly help the development effort. But unfortunately this is best done on new projects where QA/testers are not backlogged with existing tasks from previous/current projects such that they are not able to participate in the next/new project, which the developers are starting on since QA is always last in the development cycle. I think one way to more effectively involve QA in the project upfront is to make the test case and plan development in tandem or immediately after product requirements, functional specs, and use cases are defined. And test development must be completed before development actually occurs. In essence, we are making QA/testers serve as QA analysts or business analysts to clarify product features, behaviors, and interactions, filling in gaps where marketing and development failed to account for. This process might also benefit from input from some customer interaction/involvement and/or involvement of the customer support team. I think it would flush out a lot of unforeseen use cases and negative uses of the product well ahead of the product being developed so that it can be accounted for during development. If QA/testers are not familiar with the product, this phase also serves as a way for everyone (QA/testers, developers, and product marketing, etc.) to be on the same page on how the product should behave, removing misunderstandings and communication overhead that would occur later in the cycle if this isn't taken cared of early on. As others have commented, this won't be a complete solution as requirements can change (or were vague or had gaps to begin with) but it will at least greatly minimize the confusion, and bugs that arise due to vague requirements or gaps that are not clarified until late in the game when brought to attention by QA/testers or customers in the field. Testing of course still has to occur once some working version of the product is available. As for the part of clarifying behavior of a GUI product, etc. when no GUI product is available, there's always "photoshoped" mockups that could be made to help facilitate describing the model of the software to be developed, which we can then use to write test cases from (we just can't automate from these fake models). Now, if only more companies can put QA first or in parallel with development rather than in the middle or near the end of the project...

United StatesDavid Luu

11/06/2009 10:39:54 PM EST

Very thoughtful article. I certainly believe that the test effort can start as soon the functional requirements are completed. Model-Based Testing methodology calls for 5 stages: modeling, test case generation, automation scripting, test execution and result analysis (http://testoptimal.com/img/process_mbt.png). It only needs the functional requirements to perform the first two stages without the software to be tested. This allows testing to go in tandum with the software design/development. Because of the approach of using model, it's much easier to make changes to the model making Model-Based Testing a good fit in agile development process.

United StatesYaxiong Lin