Calibrated Metrics Improve Cost Estimating
By SD Times News Team
December 1, 2002 —
(Page 1 of 1)
General Dynamics Electronic Systems-as with other companies engaged in large-scale software and systems development-has grappled with the daunting "voodoo economics" of accurately estimating and controlling costs in the software development business. Yet, the company has evolved since the cost-overrun days of the 1970s and has undertaken many initiatives to improve financial projections that facilitate the delivery of high-quality systems to their customers, on time and on budget.
Major improvements came about in the 1980s as Electronic Systems, previously part of GTE Government Systems, developed a formal process for collecting and harnessing historical financial and program data to more accurately project and control costs. Over the past two decades, the company has used software metrics to dramatically improve Electronic Systems' ability to produce accurate software cost estimates and control costs throughout projects' life cycles.Although critical to mission success, the job of estimating software costs is complicated, because the processes and tools used to develop software are continually evolving. The development process is complicated by frequent changes in methodologies, languages and standards. And, the system requirements, staff and resources are ever-changing.Prior to the 1970s, the process of estimating development costs was considered a black art, and cost overruns on the order of 100 percent were common. We first tried metrics in the early 1970s with the creation of a set of "Blue Books," which included several volumes that specifically addressed standards for software and systems development processes, project planning, tracking and metrics reporting. This approach, though useful, was limited by the broad mix of different projects handled by the organization. Our projects range from the very large, over 1 million lines of code, to the very small, under 2,000 lines of code. The company also handles very formal projects, such as those with full DoD-STD-2167A, as well as very informal projects based on best commercial practices. The relatively immature metrics used at this time were unable to capture these differences.In 1981 Barry Boehm's landmark book "Software Engineering Economics" had a major impact on the company's processes. This book was one of the first to quantify the many factors that influence productivity of a software program. Some of the factors highlighted by Boehm included the experience level of the analyst writing the initial requirements, the skill of the programmer performing the implementation, the programming language used in the application, and whether or not it was necessary to rehost to a target environment. Our engineers used spreadsheets to develop models that, almost from the beginning, significantly improved the accuracy of the estimation process. By adjusting model parameters, one could bound the estimates, and thus the risks, associated with a software program. It soon became apparent that more historical data would be critical to improving the accuracy of these models, and the organization instituted a formal process for collecting this information.ROLE IN PROJECT CYCLEToday, we use metrics from the proposal phase to the post-development phase of each major project. During the proposal phase, historical data feeds the overall software estimating process. When new metric data is introduced, the estimate is continually updated. Also, during the proposal phase, comparison data is used to validate estimates. For example, a historical comparison that tracks productivity as a function of project size can be used to establish boundaries for estimating effort, schedule and staffing. The result? Except in rare circumstances, the organization's estimate of productivity rates on any one project is likely to be within 20 percent of the organization's actual average productivity rate, when normalized for size.During the program execution phase, planning is generally based on historical metric data and baseline estimates. The organization has used trend data for estimating future programs. Tracking and oversight are also based heavily on metrics, and they provide early signs of problems and help evaluate risks in costs, schedules and the like. We also use a monthly software program review metrics chart to ensure consistency in data collection. The program review process allows for timely management oversight of projects' overall progress.During the post-development phase, recently completed programs are analyzed. Actual data for such things as size, effort, schedule, staffing, defects and defects-fixed rates is collected. The final project data is then remodeled and compared with the original project estimate to assess what went right or wrong with the project and to capture "lessons learned." Recommendations are then fed into the overall estimating process to improve future estimates, and the historical data is added to the corporate metrics database.SPREADSHEETS VS. TOOLSSpreadsheet-based models dramatically improved our cost-estimation accuracy in the 1980s, but these models had their limitations. For example, they are inherently vulnerable to errors (e.g., data-entry errors) and this vulnerability increased along with the complexity of the model. Simply entering a number in the wrong cell could throw off the cost estimate for a large program by millions of dollars. Spreadsheets also didn't account for uncertainties. For example, what would the financial impact on a program be if we decided to follow a model that had a higher level of formality? Would it increase costs by 5 percent or 30 percent? Spreadsheet models could not manage these what-if scenarios.In the late 1980s, our engineers began using commercial cost-estimation tools, such as Galorth's SEER-SEM. But there's more to estimation than using commercial software: Calibrating the results of a commercial cost-estimation tool to actual program data can substantially improve the estimate's accuracy. Two important parameters for the calibration process are the levels of formality for both the specifications and the testing process. In general, the more formal the specifications and testing are, the lower the productivity!For example, a very formal program run to military standards might require high-level designs, detailed designs and test plans to be presented at stand-up presentations. These meetings might require a week of preparation and involve action items from the presentation that would have to be addressed before embarking on the next step. A less formal approach-requiring perhaps only four hours of preparation-might simply involve technical interchange meetings where the project status is explained.Calibration involves specifying the level of formality of a particular project so it matches the universe of projects contained in the cost-estimation tool's database. By evaluating previous programs, estimators can provide optimum settings for these parameters. Typically, program managers fill out forms that address each parameter contained in a model. Estimators work with program managers to compare these parameters with those parameters that have been calibrated for previous projects. Through repetition, the model provides near-perfect results. This process has made it possible to achieve highly reliable results on nearly every recent project-a level of accuracy considered an impossible dream only a decade ago.
Share this link: http://sdt.bz/26894
Most Read Latest News Blog Resources
Zeichick’s Take: Radio moves from analog waveforms to digital packets
Streaming radio highlights the need for streaming applications to be designed to take up as little bandwidth as possible
|
|
Taking enterprise architecture to the business side
Startup Corso is bringing out a cloud-based planning platform that ties into business plans
|
|
Appcelerator Acquires Cocoafish to Add Instant Mobile Cloud Capabilities to its Industry Leading Titanium Platform
Appcelerator Offers Messaging, Social, Location and Storage Mobile Cloud Services to All Mobile App Publishers
|
|
ComponentOne Releases a Collection of 40+ UI Widgets Powered by HTML5 and jQuery
ComponentOne has announced the 2012 release of Wijmo: a kit of UI widgets for HTML5 and jQuery development
|
Taking enterprise architecture to the business side
Startup Corso is bringing out a cloud-based planning platform that ties into business plans
|
|
Top five apps to manage your workload
Web applications offer new ways to track your “to-do” lists
|
|
Not so fast when it comes to testing in the cloud
Developers face outsourcing, virtual lab management and mobile devices as obstacles
|
|
Xceed releases UX-focused suite for Microsoft’s WPF
"Blendables" helps match user experiences to developer visions
|
Are you at risk for burnout?
Burnout is a severe problem and it can strike at any time. Here's how to tell if you are nearing the edge.
|
|
Agility, mom, and apple pie
If we're to evaluate the state-of-the-art in software development, we should start with the values espoused in the Agile Manifesto.
|
|
RIM woos developers with free tablet
How do you get more apps ported to the BlackBerry PlayBook? By giving every developer a free tablet, of course!
|
|
GitHire: Use Headhunters to Find Your Perfect Programmer
Are you a hiring manager tired of scouring the job boards? Check out this new service that will find 5 people interested in your jobs.
|
The Hidden Costs of Software Licensing
Moving beyond paper-based software licensing to more flexible, software-based licensing is a business decision. There is a growing trend tow...
|
|
Case Study: You May Need a Development Mechanic
As a contractor for a major financial player in Germany, SOBEGE, a German-based consultancy specializing in embedded IT and web services, wa...
|
|
Ensuring Software Quality at a Major International Bank
One of the world’s leading international banks has adopted AgitarOne technology for delivering generated unit tests for their Java software...
|
|
Load Testing Adobe Flex Applications
Adobe Flex applications may be different from applications you’ve worked with before. For classic HTML web applications, the server does all...
|