Calibrated Metrics Improve Cost Estimating

By SD Times News Team

December 1, 2002 —

General Dynamics Electronic Systems-as with other companies engaged in large-scale software and systems development-has grappled with the daunting "voodoo economics" of accurately estimating and controlling costs in the software development business. Yet, the company has evolved since the cost-overrun days of the 1970s and has undertaken many initiatives to improve financial projections that facilitate the delivery of high-quality systems to their customers, on time and on budget.

Major improvements came about in the 1980s as Electronic Systems, previously part of GTE Government Systems, developed a formal process for collecting and harnessing historical financial and program data to more accurately project and control costs. Over the past two decades, the company has used software metrics to dramatically improve Electronic Systems' ability to produce accurate software cost estimates and control costs throughout projects' life cycles.Although critical to mission success, the job of estimating software costs is complicated, because the processes and tools used to develop software are continually evolving. The development process is complicated by frequent changes in methodologies, languages and standards. And, the system requirements, staff and resources are ever-changing.Prior to the 1970s, the process of estimating development costs was considered a black art, and cost overruns on the order of 100 percent were common. We first tried metrics in the early 1970s with the creation of a set of "Blue Books," which included several volumes that specifically addressed standards for software and systems development processes, project planning, tracking and metrics reporting. This approach, though useful, was limited by the broad mix of different projects handled by the organization. Our projects range from the very large, over 1 million lines of code, to the very small, under 2,000 lines of code. The company also handles very formal projects, such as those with full DoD-STD-2167A, as well as very informal projects based on best commercial practices. The relatively immature metrics used at this time were unable to capture these differences.In 1981 Barry Boehm's landmark book "Software Engineering Economics" had a major impact on the company's processes. This book was one of the first to quantify the many factors that influence productivity of a software program. Some of the factors highlighted by Boehm included the experience level of the analyst writing the initial requirements, the skill of the programmer performing the implementation, the programming language used in the application, and whether or not it was necessary to rehost to a target environment. Our engineers used spreadsheets to develop models that, almost from the beginning, significantly improved the accuracy of the estimation process. By adjusting model parameters, one could bound the estimates, and thus the risks, associated with a software program. It soon became apparent that more historical data would be critical to improving the accuracy of these models, and the organization instituted a formal process for collecting this information.ROLE IN PROJECT CYCLEToday, we use metrics from the proposal phase to the post-development phase of each major project. During the proposal phase, historical data feeds the overall software estimating process. When new metric data is introduced, the estimate is continually updated. Also, during the proposal phase, comparison data is used to validate estimates. For example, a historical comparison that tracks productivity as a function of project size can be used to establish boundaries for estimating effort, schedule and staffing. The result? Except in rare circumstances, the organization's estimate of productivity rates on any one project is likely to be within 20 percent of the organization's actual average productivity rate, when normalized for size.During the program execution phase, planning is generally based on historical metric data and baseline estimates. The organization has used trend data for estimating future programs. Tracking and oversight are also based heavily on metrics, and they provide early signs of problems and help evaluate risks in costs, schedules and the like. We also use a monthly software program review metrics chart to ensure consistency in data collection. The program review process allows for timely management oversight of projects' overall progress.During the post-development phase, recently completed programs are analyzed. Actual data for such things as size, effort, schedule, staffing, defects and defects-fixed rates is collected. The final project data is then remodeled and compared with the original project estimate to assess what went right or wrong with the project and to capture "lessons learned." Recommendations are then fed into the overall estimating process to improve future estimates, and the historical data is added to the corporate metrics database.SPREADSHEETS VS. TOOLSSpreadsheet-based models dramatically improved our cost-estimation accuracy in the 1980s, but these models had their limitations. For example, they are inherently vulnerable to errors (e.g., data-entry errors) and this vulnerability increased along with the complexity of the model. Simply entering a number in the wrong cell could throw off the cost estimate for a large program by millions of dollars. Spreadsheets also didn't account for uncertainties. For example, what would the financial impact on a program be if we decided to follow a model that had a higher level of formality? Would it increase costs by 5 percent or 30 percent? Spreadsheet models could not manage these what-if scenarios.In the late 1980s, our engineers began using commercial cost-estimation tools, such as Galorth's SEER-SEM. But there's more to estimation than using commercial software: Calibrating the results of a commercial cost-estimation tool to actual program data can substantially improve the estimate's accuracy. Two important parameters for the calibration process are the levels of formality for both the specifications and the testing process. In general, the more formal the specifications and testing are, the lower the productivity!For example, a very formal program run to military standards might require high-level designs, detailed designs and test plans to be presented at stand-up presentations. These meetings might require a week of preparation and involve action items from the presentation that would have to be addressed before embarking on the next step. A less formal approach-requiring perhaps only four hours of preparation-might simply involve technical interchange meetings where the project status is explained.Calibration involves specifying the level of formality of a particular project so it matches the universe of projects contained in the cost-estimation tool's database. By evaluating previous programs, estimators can provide optimum settings for these parameters. Typically, program managers fill out forms that address each parameter contained in a model. Estimators work with program managers to compare these parameters with those parameters that have been calibrated for previous projects. Through repetition, the model provides near-perfect results. This process has made it possible to achieve highly reliable results on nearly every recent project-a level of accuracy considered an impossible dream only a decade ago.

Share this link: http://www.sdtimes.com/link/26894

Integration Watch: Using metrics effectively Andrew Binstock goes over the best metrics for application life-cycle management

CAST automates function point analysis The company says it has solved the complexity of measuring software function points, potentially saving time and money

Does Windows cost Microsoft opportunities? Microsoft's decision to focus .NET on Windows is stifling innovation due to the specter of intellectual property lawsuits

Add comment

Name*
Email*
Country

Comment
Preview

News on Monday
more>>
SharePoint Tech Report
more>>

Download Current Issue
ISSUE 3/15/2010 PDF

Need Back Issues?
DOWNLOAD HERE

Receive the print Edition?

Google Code turns 5
Google Code Turns 5, and adds a Paxos Algorithm to make the system more stable and reliable.
03/17/2010 11:16 AM EST

Test your Visual Studio 2010 know-how
Microsoft is offering free beta certification exams for Visual Studio 2010.
03/17/2010 11:08 AM EST

Microsoft lifts the hood on IE9
Microsoft is previewing IE9.
03/16/2010 01:10 PM EST