Past As Predictor And Earned Value - Part 2
A Letter of Clarification from Wayne Abba Vice President
Integrated Management Services
Dekker, Ltd.
Bill Duncan Comment
1. History of Earned Value. The technique was NOT developed by the USDoD in the 1950s and 1960s. It was developed by some Industrial Engineers way back in the 1930s as an alternative to simple budget-vs-actual reporting. Although there wasn't much of a formal discipline of project management in those days, there obviously were projects -- much of President Roosevelt's economic recovery program was driven by federal spending on major construction projects. EV remains superior to BvA.
Wayne Abba Reply
DoD documented and formalized the technique, based on research performed in the 60s by the Air Force at Boeing and other defense contractors. The criteria-based approach to management oversight, as opposed to prescriptive "how to" requirements (e.g., PERT), was a significant improvement that has stood the test of time for more than 35 years. Contemporary EVM is a product of defense experience, increasingly adopted by others.
Bill Duncan Comment
2. USDoD Version of Earned Value. We should be careful to separate the USDoD
version of this technique from the basic concept. Lumping all forms and implementations
of the technique into one bucket is dangerous and confuses the issue. The
USDoD approach was designed for
MAJOR weapons procurements. The last time I checked, the USDoD procedures
exempted a variety of projects from its EVM requirements, including what it
considers smaller projects and most R&D projects.
Wayne Abba Reply
This is incorrect. The US DoD approach had two main thrusts: "full" EVM (formerly known as Cost/Schedule Control Systems Criteria or C/SCSC) for major contracts and the Cost/Schedule Status Report (C/SSR), a much less disciplined methodology tailored for smaller contracts (couple million and up). The main difference was the rigor used to determine planned and earned value. DoD does not exempt smaller projects from EVM techniques and never has. As for R&D projects, EVM is exempted only for conceptual development - "inventive" processes if you will. EVM works well for R&D and is mandatory when development hardware/software and early prototypes are involved. The emphasis on implementation thresholds has given way to judgment based on a contract's type, complexity, and risk.
Bill Duncan Comment
3. USDoD survey results. The USDoD maintains an extensive database of historical
results. Their results show that if a project's CPI is 0.90 or worse (i.e.,
10% over budget) after 15% of the budget has been expended (i.e., 1/6 of the
way through the project), the project's CPI
never improves. This result is often misquoted (even by such as Quentin Fleming)
as "the CPI never gets better." In addition, a large percentage
of the USDoD database includes cost-plus projects where there is limited incentive
for the contractor to improve the CPI. In my
opinion, these results are illuminating but not generalizable.
Wayne Abba Reply
Most of the research was done by the Air Force Institute of Technology - see the PMI College of Performance Management website http://www.pmi-cpm.org/ for a link to the comprehensive bibliography of EVM literature, hosted by Prof. Dave Christensen at Southern Utah University. Dave was a professor at AFIT who directed the research, much of it under my sponsorship (we didn't have the resources in the Pentagon to do the work). I worked from 1972-99 as the DoD's senior program analyst for contract performance management. Quentin's shorthand is a fair representation of Dave's oft-quoted 10% rule -- the CPI from an early point forward does not change by more than 10% and usually gets worse.
A large percentage of the database includes fixed-price incentive contracts as well as cost-plus and cost-plus incentive. Even firm-fixed price contracts use EVM when in the PM's judgment the visibility is needed (as for example when a company is making a large investment of its own resources or when the contract is crucial to the overall program).
Bill Duncan Comment
4. Other survey results. The only other detailed study that I have seen comes from an Israeli electronics contractor. Their results (covering about a dozen projects in the late 1980s and early 1990s) showed that their CPI was a reliable predictor of total project costs after 25% of the project's funds had been expended. I consider these results more generally applicable because these projects were done under a fixed price contract.
Wayne Abba Reply
Cost-type or fixed price is not the point. The data were scrubbed to eliminate abnormal situations such as overtarget baselines, leaving a statistically significant group of contracts including all military services and commodities - airplanes, ships, missiles, armored vehicles, electronics, software (a large part of most DoD developments) and more. The results proved insensitive to cost vs. fixed price contract type or any other category such as commodity or buying agency. In other words, the results indeed are generalizable.
But generalizing is not the point either. EVM's strength lies in its ability to show a project team how it is doing based on its own plans, allowing the team to make course corrections or to replan as necessary and to provide defensible estimates to management.
Bill Duncan Comment
5. Impact of CPM on EV projections. Although activities with float can appear
to distort the numbers, I maintain that these kinds of variances still require
analysis. If a non-critical path activity is behind schedule, the experienced
PM generally wants to know why: presumably there was a reason for scheduling
that activity at that particular time. If it is not on track, the delay could
propagate throughout the project and cause problems. There is nothing that
says
the PM must implement corrective action in response to every variance, but
someone should certainly be looking at each variance to assess the need for
action.
Wayne Abba Reply
Of course variances require analysis. Variances associated with CP activities obviously require more immediate attention, but activities that threaten the CP do too. Contemporary EVM software directly associates EV variances with the schedule to simplify the schedule analysis task.
Bill Duncan Comment
6. Projection techniques. There are a variety of ways to use EV data to project the project's cost at completion. A simple straight line projection is recommended ONLY when the project management team feels that the project's future activities are "similar" to its past activities. "Similar" could mean that it is the same kind of work, was estimated in the same way, will be performed by people with similar skills, or has similar degrees of uncertainty. Both "A Guide to the Project Management Body of Knowledge" and the USDoD EVM guidance documents explain a variety of alternative projection techniques and detail when each is appropriate. EVM is more sophisticated than simple straight line projection. It also relies heavily upon trends rather than simple point-in-time projections. Trend analysis improves the reliability of the projection.
Wayne Abba Reply
EVM research has shown generally at what project phases various algorithms work. The cumulative cost performance index, for example, provides a reliable "floor" estimate early on while other techniques relate better to other project phases. However, those are only sanity checks. The "real" estimate at completion is always the project team's responsibility. The independent EAC techniques are valuable for oversight organizations and for portfolio analysis. The best analysis in my experience is graphical - watch the trends and see how they interrelate. Does planned value accurately depict planned resource availability? Does the plan make sense when projected to the end date? How are variances doing? What is the efficiency of work performed and how does it relate to the efficiency required to achieve the estimate at completion?
Bill Duncan Comment
7. Projections generally. No process is perfect. No intelligent, rational
person expects that EV projections or any other kind of projections will always
be able to predict the future exactly and
precisely. Remember that the term is ESTIMATED cost at completion. But that
doesn't mean that imperfect projections aren't valuable. If I think I'm doing
okay, and the numbers say I'm not, I better be able to explain why (or else
I better change my mind).
8. Use of SPI. SPI is less reliable than CPI in predicting future performance because SPI can almost always be improved by spending more money. However, an SPI of less than 1.0 does mean that you are behind schedule, and an SPI that is trending down means that you are getting farther and farther behind schedule. That information should be useful.
Wayne Abba Reply
An SPI of less than 1.0 does not necessarily mean you are behind schedule. If planned value is tied to early start dates, it is entirely possible to be behind in EVM terms and ahead on CPM. Not to mention the washout effect of variances. That said, the SPI indeed is a reliable early warning indicator when early, unfavorable and large. Hard to quantify that but I know it when I see it on performance trend graphs and so can anyone who pays attention.
Bill Duncan Comment
9. Alternatives. I have yet to see a viable alternative that integrates cost and schedule performance. Donna's rolling forecast of cost-to-complete can be viable in the hands of a skilled and experienced PM, but it can also be dangerous. I know of one commercial software developer that drove itself into bankruptcy because its rolling forecasts were delusional -- but believed by management. In this case, the EV projections showed a 400% overrun.
Wayne Abba Reply
Rolling forecast of cost-to-complete sounds like a variation of the old PERT-COST technique. Didn't work then for complex projects and won't now because it relies on the accuracy of estimates to complete. EV on the other hand provides an objective measure of progress. As you say, the only viable approach to integrated cost/schedule/technical performance management.
Bill Duncan Comment
10. A rose by any other name. The approach used to be called EV Analysis. Its proponents started calling it EV Management several years ago because (in my opinion) the USDoD PMs saw "analysis" as a staff function. Personally, I prefer the old name because it emphasizes that the technique only provides information; that it is still up to the PM team to figure out what to do with the information.
Wayne Abba Reply
EVA (EV Analysis) is a relatively new UK term. EVM used to be called C/SCSC. My boss and I changed it to EVM (and even better, Integrated Program Management) to emphasize its role in an integrated management process and to get rid of the emphasis on "cost" and the resulting unfortunate identification with bean counters.
Bill Duncan Comment
11. Silver bullet. There ain't one. EV is just one tool among many. It can be highly effective when applied in the proper environment (larger projects with relatively stable scope) with knowledge and skill. Once again, let's not confuse the tool with the application. Let's not throw out the baby with the bath water.
Wayne Abba Reply
EVM is scalable - the smallest application I've seen is the 2-week shutdown of a paper mill for scheduled maintenance. EV was run on an hourly basis, very simple. Provided objective progress status and timely ability to reorient resources, for example, when a needed part was not available. The mill came back up and operated at unprecedented efficiency, every task having been performed. Better yet, the managers said they could sleep at night knowing where they stood... a good lesson for all of us. For large projects with unstable scope, EVM is an excellent way to inject a reasonable degree of disciplined management. The alternative is ... what? Chaos?
Reference 1: For a comprehensive literature of Earned Value see http://www.cpm-pmi.org/
About the Author
Wayne
Abba was appointed Vice President for Integrated Management Services for
Dekker, Ltd., a leading provider of project management software and decision
support services, in August 1999 following a distinguished career with the
United States government. In 1999, he received the Project Management Institute's
Distinguished Contribution Award for advancing project management in the
public and private sectors. At Dekker, Ltd, Wayne continues his leadership
through development of new management and analysis tools. He can be reached
at 703-391-7256 or
w.abba@dtrakker.com.