Process measures to improve R & D scheduling accuracy - HP's Roseville Networks Division's research and development project management - technical

Process Measures to Improve R&D Scheduling Accuracy

PRODUCT DEVELOPMENT TEAMS at Hewlett-Packard typically commit to a project completion date at the conclusion of requirements definition, just before implementation. Meeting the commitment data has been a perennial challenge, particularly for software engineering teams. Last eyar, the Roseville networks Division R&D laboratory began a major campaign to improve the accuracy of project schedules. Key to the effort is a view of scheduling as an ongoing process subject to continuous, objective measurement.

Roseville Networks Division is a member of HP's Information Network Group, which supplies networking products for technical and commercial computer systems. At any time, approximately two dozen R&D projects are in progress at RND. Each project may suffer one or more unanticipated setbacks durign the course of development. Although the particular events affecting a single project cannot be predicted, the cumulative effect of all sources of project delay can be measured and used to improve the accuracy of future schedule estimates. Since conventional measures of scheduling accuracy, such as DeMarco's estimation quality factor, or EQF,.sub.1 are not suitable for real-time measurement of a set of project schedules, we developed two new process measurements that provide sensitive and timely indexes of project progress.

One of these measures is a modified form of DeMacro's EQF. By recording schedule adjustments on a month-by-month basis, we are able to make an accurate running estimate of the aggregate EQF of all lab projects. Another measure is project progress rate, which provides an ongoing prediction of the average actual project durations normalized by the average of the estimates. These metrics provide rapid feedback of schedule estimators on the accuracy of their project plans. Since we began the measurements a year ago, the effective [projected] EQF of the RND lab has improved by a factor of three. Our average lab-phase schedule duration overrun has been reduced from 70% to 20%.

This paper describes the derivation of these metrics and their application in the Roseville Networks Division lab.

Process Perspectives

Like other HP entities, RND uses formal software and hardware life cycles to define the internal process steps required for new product development. The most important milestones of the process are the investigation-to-lab (I-L) checkpoint and the manufacturing release (MR) checkpoint.

I-L marks the transition from the requirement definition and project planning phase to the implementation (lab) phase. At this checkpoint, the project manager is expected to have prepared a detailed project plan with dates for the intermediate milestones and for the MR milestone. This is the time at which the formal commitment is made by the project team to deliver on a particular date.

MR is the official end of the development effort, when all lab engineering and beta test work are complete. The product is suitable for customer shipments at this time.

Of the approximately two dozen projects active at any given time at RND, somewhat less than half are in the investigation phase and the remainder are in the lab phase. This is a large enough number that the projects can be considered collectively from a process perspective. Process measurements of scheduling accuracy can be applied and process improvements identified that benefit the entire organization.

The central issue in R&D for delivering on commitments is the match between actual lab-phase durations and I-L estimates of duration. Historically, the ratio of actual project durations to the I-L estimates has been about 1.7 at RND. Our process objective is to improve the ratio to 1.1 or better.

Process Measurements

Techniques in common use within HP for project tracking work well when applied to individual projects. A Brunner diagram, for examples, gives a good picture of time and effort expenditures in comparison with estimates as a project progresses. But it is difficult to combine information from Brunner diagrams to get an overall sense of how well an organization is managing projects.

Likewise, DeMarco's EQF has been used to evaluate the quality of schedule and effort estimates for individual projects. DeMarco suggests that project EQFs be averaged to obtain an overall value to represent an organization. Unfortunately, the average can be made useless by the presence of one well-planned project with very high EQF. EQF has a second drawback as a process measurement. The metric is normally calculated upon project completion, so it does not provide rapid feedback to aid improvement efforts while projects are in progress.

These considerations prompted the Roseville Network Division to develop new metrics to monitor the scheduling performance of the lab. For the past year, we have applied the process EQF (PEQF) and project progress rate measures to all active lab-phase projects with good results. Although our focus has been on tracking project calendar time, the metrics could be applied equally well to tracking engineering effort.

A feature of the RND lab environment has made data collection for the metrics relatively simple. Each month, all project managers are required to enter schedule updates in a central data base for status reporting purposes. The data base currently contains information covering over four years of lab history. Data from the status reports is entered once a month into a spreadsheet for the production of the metric graphs.

Process EQF

Process EQF (PEQF) is a modification of DeMarco's estimation quality factor specifically developed for application to a set of ongoing projects rather than to individual projects. PEQF provides a sensitive real-time index of scheduling accuracy that is free of the potentially misleading effects of EQF averaging. DeMarco defines EQF as a ratio of areas on a chart of estimates versus time maintained for a project. Referring to Fig. 1, let E.sub.i,1 be the first or I-L estimate of, for example, L-phase project duration for project i. E.sub.i,k is the estimate for the kth interval (of a total n equal intervals) during the project. A.sub.i is the corresponding actual duration recorded for project i on completion.

EQF for project i is then:

Consider the EQF for scheduling of a project i that experiences a constant rate of slippage. E.sub.i,1 and A.sub.i are the estimated and actual lab-phase durations, respectively. The history of this special case is diagrammed in Fig. 2.

The average rate of slip d.sub.i is just the slope of a line drawn from (0,E.sub.i,1) to (A.sub.i, A.sub.i). This line forms a right triangle with the vertical axis, and it is clear that the error area can be represented by the triangle area. Therefore,

Substituting the slope in the above equation we get

Note that when the slip rate is constant we do not need to know the initial estimate E.sub.i,1 or the actual duration A.sub.i to calculate EQF.

Definition of PEQF

Consider now a set of projects collectively slipping at an average rate , where the average rate of slip is approximately constant. This situation is diagrammed in Fig. 3.

In analog with the case described above, we define PEQF as where = cumulative slip for all projects in the set / cumulative elapsed project time

Fig. 4 illustrates the general case in which the average rate of slip varies slowly with time. A projection of process EQF can be made at each calendar interval k using the average slip rate at k. Thus, where = sum of project slip times in interval k / sum of project elapsed times in interval k.

As shown in Fig. 4, the projection is made with a tangent line to the curve at k. Observe that the tangent line forms a right triangle within the box in the diagram. It can be seen that the PEQF projection has a geometric interpretation as a ratio of areas, which is similar to the original definition of EQF.

PEQF Results

Fig. 5 is a plot of actual values of cumulative slip versus cumulative elapsed project time for L-phase projects in the Roseville Networks Division lab. The curve includes nearly four years' project history at RND and is indeed slowly varying like the curve of Fig. 4. The interval k in this plot is one month.

Fig. 6 is a plot of historical PEQF.sub.k values for the RND lab. For this diagram the interval has been increased to 3 months to reduce the effect of randon fluctuations in the timing of schedule adjustments. This chart has been available to lab management each month since May 1986. Before that, the average PEQF was about 4.0. Since then PEQF has improved substantially, with one recent peak value exceeding 22.

Project Progress Rate

Although PEQF is useful, it, like the original EQF, is an abstract number having no simple or direct connection to schedule performance in management's terms. Management is less concerned with a ratio of areas than with the idea of completing a project on the commitment date.

RND has found it instructive to watch the ratio of the actual durations of projects to the I-L estimates of duration. A two-month slip represents a much larger scheduling error for a four-month project than for an 18-month project. Normalizing the actual durations by the estimates allows projects of differing lengths to be compared directly on a percentage basis.

This ratio is also helpful to evaluate the scheduling accuracy of a set of projects. Fig. 7 is a scattegrgram of estimated project durations versus actual durations. The slope of a best-fit straight line through the origin and the data points is given by where A.sub.i is the actual duration of the ith project and E.sub.i is the estimate. (Note to statisticians: Although this is not a least squares lines, it is an unbiased estimator. The variance is somewhat larger than with least squares, but a test on RND project data shows the difference to be insignificant.)

Unfortunately, m is only available after a number of projects have been completed and hence is slow to respond to process improvements. What is needed is a real-time predictor of m that is sensitive to process changes as they occur. Project progress rate is just such a predictor.

The estimator of m we require can be developed as follows: where [delta].sub.i is the total slip of project i: [delta].sub.i = A.sub.i - E.sub.i.

To estimate m at time interval k we need an estimate of at k. Let

Substituting in the expression for m above, we define the project progress rate at interval k as

Project progress rate is inversely related to m rather than directly for psychological reasons. We prefer a metric that increases when things get better. "Perfect" on this scale is a value of 1.0 sustained over time. Lower project progress rate values correspond to higher projected values of m and hence longer average schedule overruns.

Project Progress Rate Results

An m value of 1.7 was calculated in mid-1986 using RND historical data from 21 completed projects. For comparison, the project progress rate metric was calculated for the same set of projects. The two results agree remarkably well; the discrepancy is just 1%. This is a strong empirical confirmation of the relationship between overall scheduling accuracy (expressed as a ratio of actual and estimated durations) and the project progress rate metric. The metric has increased through the year and now forecasts an m value of about 1.2 for currently active L-phase projects. If this performance is sustained, RND will deliver the next generation of products with an average of 20% or less schedule overrun.

Fig. 8 is a chart of the project progress rate metric as it is used at RND. The chart is updated monthly to show I-phase projects (reflecting slips of the promised I-L dates), L-phase projects (reflecting slips in the MR dates), and the combination of the two. To reduce fluctuations, the interval size for metric calculation is three months. A lab with a smaller number of active projects may need to select a longer interval to ensure a clean and readable graph.

Even with three-month intervals, there is noise in the RND graph. Fig. 9 shows the L-phase data using a 12-month interval. This chart reveals a long-term trend of declining scheduling accuracy which was not apparent in the previous graph. A dramatic reversal in this trend began at the time we made project progress rate a key management tool in the effort to improve scheduling accuracy in R&D.

Conclusions

This paper has outlined the derivation and application of two process metrics that monitor overall scheduling accuracy in an R&D organization. The two metrics are quite similar; they both report average scheduling performance across many projects in a timely manner. The similarity has a deeper origin, however. Both the process EQF and the project progress rate measures are based on the average slip rate, d.sub.k. Thus the choice between PEQF and project progress rate is primarily a matter of style. PEQF is recommended for organizations that have internalized EQF as a project estimation measure. Like EQF, PEQF tends to magnify small differences as scheduling perfection is approached. Values nearing infinity are theoretically possible. By contrast, project progress rate has a more uniform scale with values approaching 1.0 as accuracy improves. Project progress rate has the additional advantage of forecasting the average percentage schedule overrun for the organization.

Project progress rate is the primary schedule management metric at RND. The metric is quite scalable, and could in principle be used at a division lab level, at a group level, or across an entire corporation.

Although the topic of this paper is the metrics themselves, the discussion would be incomplete without a mention of the actions that have brought about substantial scheduling improvements at RND. The metrics have made scheduling accuracy much more visible to management. Management attention and emphasis on accuracy have caused project planners to devote more effort and care to scheduling, with beneficial results. Project managers have received training in scheduling and estimation. A personal-computer-based project management program has been introduced and is now in use in the RND lab. To ensure that project managers benefit from each other's experience, we have begun to conduct peer reviews of proposed schedules. A task force of project managers has been formed to discuss common causes of schedule error and identify corrective measures.

Although our overall schedule accuracy has improved, we feel that further gains are desirable. We have begun to understand that the significant sources of delay include unforeseen tasks, changing project objectives, coordination with outside groups, and shortages of tools. Our next challenge is to reduce or eliminate these sources of delay and promote overall gains in productivity. The measures of scheduling accuracy described here will provide rapid feedback on the effectiveness of these improvements as they are made.

COPYRIGHT 1988 Hewlett Packard Company
COPYRIGHT 2004 Gale Group