"Inadequate measurement is itself a root cause for many of the most severe problems of the software industry including low productivity, false productivity claims, low quality, long schedules, missed schedules, excessive time to market, inadequate estimating, friction with senior management, friction with users, low user satisfaction, canceled projects, inadequate methodologies, inadequate tools, poor organization structures, inadequate specialization, inadequate capital investment, and inadequate office environments…Companies that have established full Applied Software Measurement programs often have improved quality by more than 60% and productivity by more than 30% within three years…" 
-- Capers Jones

Practice Description

All world-class software organizations that the SPMN has visited have an extensive metrics program for early indicators of evolving problems on an ongoing project and for measuring the effectiveness of management and technical practices for improving the bottom line. Almost all the useful software metrics can be collected at very little additional cost as part of implementing some best management or technical practice. The value from a metrics program comes from the actions taken as a result of metrics. Therefore, collecting metrics solely for the sake of collecting metrics is not a best practice and is, in fact, counterproductive. Of key importance in an effective metrics program is having predefined thresholds for metrics values trigger predefined actions when measured metrics values are outside their threshold values.

Practice Essentials

• Collect metrics on:
    
  • Early indicators of potential problems 
  • The quality of the products 
  • The effectiveness of the processes 
  • Conformance with process details 
  • Data on which to base future cost estimation
     
• For each early warning metric, define threshold values for reporting potential problems to different levels of management.
   
• Early warning metrics from all organizations involved in the program should be reported to the buyer's PM in monthly status report.
   
• Early warning metrics reported to project managers should be based on data current to at least one week before the date the report is submitted.
   
• Metrics data should be easily available to all persons on the development/sustainment team.
  
  
• The metrics selected to be a project’s main indicators of health or dysfunction should provide program wide visibility into the project’s status and an accurate comparison of progress against the plan.
  
  
• At a minimum, metrics should measure and track schedules, budgets, productivity, defects, quality, and risk. In addition to this basic set, metrics should be added, as necessary, to monitor specific project concerns.
   

Implementation Guidelines

• Every project will have a project plan with a detailed activity network that defines the process the team will follow, organizes and coordinates the work, and estimates and allocates cost and schedule among tasks. The project plan will include adequate measurement in each of these five categories.
   
  • Early indications of problems 
  • The quality of the products 
  • The effectiveness of the processes 
  • The conformance to the process 
  • The provision of a basis for future estimation of cost, quality, and schedule
     
• Metrics will be sufficiently broad-based. Data will be collected for each process/phase to provide insight into the above five categories.
   
• To use these metrics effectively, thresholds will be established for these metrics. These thresholds will be estimated initially using suggested industry norms for various project classes. Local thresholds will evolve over time, based upon experience. Violation of a threshold value will cause further analysis and decision making.
   
• Examples of data, initial thresholds, and analysis of size, defect, schedule, and effort metrics can be found at: http://www.qsm.com.
• Continuous data on schedule, risks, libraries, effort expenditures, and other measures of progress will be available to all project personnel (supplier/developer and acquirer) along with the latest revision of project plans.
   
• Process-effectiveness metrics should include:
   
  • Defect removal efficiency 
  • Percentage of development cost due to rework 
  • Defect leakage through inspections 
  • Initial size estimate and size of delivered software 
  • Initial estimates of changes to reuse/COTS/GOTS code and actual changes made
     
• Collect and report at least the early warning metrics from the SPMN Control Panel.
   
• Early warning metrics should include schedule compression.
   
• Threshold values for early warning metrics should be defined based on industry norms and impact on project success.
   
• Quality metrics should include code complexity and delivered defect density.
   
• Collect and report the number of defects found in each formal inspection and leakage of defects through inspections.
   
• Data to support future cost estimation should include:
   
  • Actual cost 
  • Actual productivity 
  • Actual size 
  • Percentage of total code that is reuse code 
  • Percentage of initial reuse code SLOC that is modified or added code to allow reuse in the new application 
  • Type of software (IS, engineering, real-time, hard real-time) 
  • Technical and management methods and processes followed 
  • Average monthly requirements and staff volatility 

Things to Watch Out For

• Every world-class software development organization visited by SPMN had a substantial metrics collection and analysis program in place.
   
• Most organizations implementing software process improvement to achieve higher SEI levels do not measure the effectiveness of their processes in improving the bottom line.
   
• Many organizations do not share the metrics that are reported to the customer with the technical staff.
   
• Many software managers have not established threshold metric values to trigger reporting.
   
• Rarely is requirements volatility measured.
   
• Rarely is staff volatility reported, and many suppliers/developers strongly resist giving the DoD program office visibility into this metric.
   
• Rarely does the SPMN see a project that computes and reports the current value of schedule compression.
   
• Rarely does the SPMN see a project reporting the number of defects that were found during each inspection that was conducted.
   
• Most projects do not collect metrics that indicate excessive rework.
   
• Earned value metrics that are reported each month are often six or more weeks out of date.
   
• The earned value Cost Performance Index (CPI) and To-Complete Performance Index (TCPI) metrics, which, when combined, give an indication of the validity of the current Cost at Completion estimate, are almost never reported.
   
• Earned value metrics are often measured by an activity network that is of such poor quality that the metrics are essentially meaningless. 

Figuring Out Where You Are - Project QuickLook

Chapter 6 contains a series of detailed questions that can be applied to any program to determine the health of its metrics-management program. These questions should be applied on a regular basis. However, there are some quick indicators that can be used to see if the process requires more in-depth examination. Specific "quicklook" questions are:

• Are metrics routinely presented to Project Management? 
• Are the metrics viewed by your staff as "just extra work"? 
• When was the last time the metrics showed an anomaly (weren't what was expected)? 
• Have threshold values been established? 

An inappropriate answer to any of these questions could indicate a more serious problem and merits further investigation.

top

---