The article “Beyond the OPM3 Hype: A Reality Check” by Craig Curran-Morton is a much needed invitation for discussion and critique of OPM3, which was published at the beginning of this year. While his article misinforms in several important ways it creates a useful, timely, and public conversation that will help current and prospective users of OPM3.
Assessment Methodology and Organizational Assessment
An initial concern with the OPM3 involves the assessment methodology. Craig notes “it is based on Yes/No answers which are very perplexing when you’re trying to conduct a comprehensive assessment of an organization’s project management practices.” Sure, they can be perplexing to anyone unfamiliar with the PMBOK Guide, and many people are not familiar with it even though more than 100,000 copies are in circulation and it is published in several languages. However, Craig’s article does not explain that the Yes/No questions are not, in fact, designed to or intended to identify bona fide capabilities and actionable improvement options. The pitfall is that s prospective users of OPM3 are likely to make this mistake easily, i.e. they may not distinguish OPM3’s Yes/No questions from a comprehensive assessment using the indices or directories of OPM3.
The “Yes/No answers” refer to 151 questions that are provided in the form of a supplementary tool on the CD that comes with the OPM3 book. The 151 questions are embodied in a rather elementary software application (with limited functionality) bundled on the CD.[1] An OPM3 assessment does not require one to use these 151 questions. But the 151 questions can be quite useful when used in the correct context. “What context?” you may ask. The 151 questions can help the user to explore whether or not it is appropriate for his or her organization to use a SUBSET of the OPM3 content in a comprehensive maturity assessment. It is very important for prospective users of OPM3 to understand this, i.e. the difference between the “High Level Assessment” that employs the 151 questions and a comprehensive “Capability Assessment” of an organization’s capabilities. The only way to determine which capabilities an organization does or does not have is to perform a Capability Assessment in which the Capabilities of a Best Practice (as outlined in OPM3’s Improvement Directory) are assessed for their factual existence as demonstrated by evidence (i.e. outcomes, key performance indicators, metrics). The 151 Yes/No questions do not do this.
OPM3 defines 586 best practices. Because it is often difficult for an organization to institutionalize a best practice instantaneously, and because the exact meaning and method for executing a best practice needs to be clear, and because OPM3 should enable users to assess and re-assess themselves without requiring the ongoing support of consultants, OPM3 defines each of its 586 best practices in terms of the constituent capabilities that are required in order to effect each best practice. In other words, each best practice is decomposed into a series of “steps” that result in the best practice. Each “step” is articulated as a capability that the organization should develop in the pursuit of best practices that represent maturity. The degree to which capabilities have been achieved is one’s level of maturity. This is why OPM3 is a “capability maturity model” or “CMM”. Unlike many CMM’s, OPM3 is modular, meaning that each best practice is separable from the whole; therefore, one assesses maturity per best practice by evaluating the capabilities that are constituent to each best practice respectively. This has the benefit of demonstrating a clear roadmap for improving the capabilities required for each best practice.
Answering the 151 questions results in a “rough and ready” overview of the perceptions of people within the target organization, perceptions regarding the current state-of-affairs concerning their own portfolio, program, and project management practices. Such results are not useful for determining if specific capabilities exist within the organization. They are not useful for planning specific improvements either. But they are useful for creating momentum and for supporting decisions regarding where to focus one’s assessment. They may help users decide to use no more OPM3 content than needed to make the few most useful improvements. In contrast to the 151 Yes/No questions, a Capability Assessment harnesses the power of OPM3's most salient and distinguishing design characteristic, which is OPM3's ability to report one's specific capabilities while simultaneously and automatically identifying one's improvement options in detail, i.e. the next Capability above the current-state Capability in the sequence for a Best Practice. The Capability Assessment as I have described it (and not the 151 Yes/No questions) was the original assessment strategy from the onset dictated by requirements derived from the market. If you want more information about the rigorous and methodical development of requirements our team used to guide the design of OPM3 using the “House of Quality” technique and over 30,000 survey responses from prospective users of OPM3, read The History of OPM3. The following summarizes the main differences between the High Level “Assessment” using the 151 questions and what I have called the “Capability Assessment,” which is referred to in OPM3 as a more “comprehensive” or “detailed” assessment process:
High Level Assessment
¨Designed to provide directional guidance to help make a decision regarding the scope the comprehensive assessment
¨Designed to establish consensus and support the Assessment / Improvement team
¨Not designed to identify bona fide Capabilities and actionable improvement options
Capability Assessment
¨The real litmus test regarding one's current state of maturity
¨Enables us to identify specific recommendations for improving Capabilities
¨Is a required step if our goal is to implement any part of OPM3 in any part of our organization in the manner OPM3 is designed to be implemented
The “High Level Assessment” does deserve criticism. The High Level Assessment (151 Yes/No questions) is a sociological exercise, merely a survey of people's perceptions. It results in an inference, which is not the same thing as an assessment of fact or evidence. Indeed, it is an inference of a very imprecise nature based on reported perceptions regarding general things. The "best practices" are the general things in question. The 151 Yes/No questions do not ask anything about the constituent capabilities of the OPM3 best practices. In fact, each of the 151 questions is a general question referring to an aspect of organizational project management that may correspond to many OPM3 best practices. For example, the first of the 151 questions is based on four OPM3 best practices, i.e. OPM3 Best Practices 1440, 1450, 1510, and 3060. Members of an organization that is going to be assessed using OPM3 may answer this first question of the 151 questions without knowing that this question implicitly refers to multiple best practices, and they may not have any knowledge of the constituent capabilities of those best practices. This is a shocking design flaw if one assumes the 151 questions were designed to determine an organization’s capabilities. It is not shocking if one understands that the correct use of the 151 questions (if one uses them at all) is to facilitate a discussion about whether or not all OPM3 best practices and their constituent capabilities should be included in one’s maturity assessment. You may perform a “High Level Assessment” using the 151 questions as a filter before you perform a more comprehensive Capability Assessment .
So many people may be surprised, unfortunately, for a couple reasons: A) the 151 questions are referred to as a “High Level Assessment” when, in fact, the word “assessment” is misleading, and B) the software that contains the 151 questions produces a series of simplistic graphics that start with an aggregate “maturity score” on a “continuum of organizational project management maturity.”[2] It’s a shame that these 151 questions were not packaged and explained well in the final product after years of developing a comprehensive model with a robust assessment methodology, even though the comprehensive model and associated robust assessment methodology do comprise the majority of OPM3 today.
Craig reports that he heard from someone from the team of volunteers that developed OPM3 that the 151 questions were added as an afterthought at the very end of the program to develop OPM3. That’s true. I was on that team up to the point in time when this afterthought became a requirement. I was the leader of the OPM3 program from kick-off through delivery of the prototype, at which point I resigned my position. Just before I resigned my position, there was much colorful discussion about a new requirement from our sponsor that OPM3 must provide some kind of overall maturity score showing an organization’s position on a “continuum of organizational project management maturity.” This was not one of the original requirements, but it was suggested that every user of OPM3 should evaluate all of the capabilities of all of the best practices. I maintained that such a requirement would defeat many of the most useful characteristics of OPM3. “Which characteristics?” . In particular, all of the relationships among best practices in OPM3 had been defined, and in turn, all of the inter-dependencies among capabilities leading to best practices had been defined in precisely the same manner that the network diagram of a project schedule defines finish-to-start dependencies leading to an end goal. Indeed, defining these relationships is what “took so long.” These relationships were defined in order to satisfy the requirements that OPM3 must be “scaleable” and “flexible.”
“How do these relationships make OPM3 scaleable and flexible?” Because the relationships are known (and defined as a modular architecture as explained above), one can select any OPM3 best practice and see its predecessors, and for this reason one may choose to focus on any combination of best practices without having to evaluate every last best practice defined within OPM3. If you’re interested only in the best practices associated with Portfolio Management, you can limit your maturity assessment to that domain. If you want to focus on Program Management, you can do that instead. If you wish to assess only Project Management, fine. Any combination of portfolio, program, and project management best practices is possible, and if you wish to compare your assessment to another organization’s assessment you need only to assess at least some of the same things in both organizations. Each organization is unique, and OPM3 gives you options because one size does not fit all.
The new requirement (which was eventually rejected) that “users of OPM3 should evaluate all of the capabilities of all of the best practices” struck me as unrealistic and unhelpful in a model that has 586 best practices and even more capabilities. I argued that this is like requiring a person on a road trip from a point of origin to one destination to consider the routes to all possible destinations. Nobody would use roadmaps if such constraints existed, or they’d just ignore the constraint altogether. The debate culminated in a decision (which I opposed) to create a “Reader’s Digest” version of the model in terms of a concise list of Yes/No questions, i.e. the 151. Having spent over 4 years developing OPM3, I wasn’t happy with this new turn of events, but in retrospect it worked out pretty well. “How can you say it worked out?!” . The 151 questions do serve the one very specific and useful purpose I explained above, i.e. they help the user to determine whether an opportunity exists to use a SUBSET of the entire set of OPM3 best practices and constituent capabilities when performing a real Capability Assessment. In other words, they help us ask the question "Do we really need to use the encyclopedic scope of OPM3 in every maturity assessment?" A variation of this question may be “Can we save time and money yet do something relevant and valuable for our own organization?” If you know what you are doing, you can complete a “High Level Assessment” using the 151 questions in one day. I assert this based on experience, as my firm specializes in conducting OPM3 assessments. One day’s work to create the possibility of preventing many additional days of less value-added work? Now that is both realistic and helpful.
Craig notes that “To the OPM3, an organization is either pregnant or it is not: It either has a specific capability or it doesn’t.” As I explained above, the 151 Yes/No questions do not ask about specific capabilities, but in fairness it is true that one should define the “business rules” that one is using when answering the 151 Yes/No questions about best practices, especially if more than one party within a target organization is answering the questions. By contrast, a “Capability Assessment” provides much more detail to take guesswork out of the process while defining shades of gray as discrete and clear degrees of maturity. A Capability Assessment uses the specific capabilities that comprise each of the best practices that one includes in one’s maturity assessment. Each capability is defined as a complete sentence, and the outcomes that one should expect from each capability are defined. Likewise, for each outcome, OPM3 lists the few “key performance indicators” that one should seek when evaluating whether an outcome has been produced by a capability. Furthermore, each key performance indicator is defined in terms of metrics. In other words, there’s plenty of information provided in OPM3 to help users determine whether or not any given capability exists. This facilitates self-assessment and takes the “fuzziness” out of the results of a maturity assessment. You don’t just find out whether you’re “pregnant,” if I may extend Craig’s metaphor. You find out the status of the zygote and its host, so to speak, and precisely what you may do next to promote development or improve. Although the 151 Yes/No questions of the High Level Assessment are imprecise (because they are concise in the extreme by design), the questions inherent to a more comprehensive Capability Assessment are precise at level of granularity that facilitates highly actionable and focused planning. I view this as a strength, not a weakness.
To the larger point, if PMI continues to call the 151 Yes/No questions a “High Level Assessment”, more complaints and problems will arise. The 151 questions are helpful but non-essential. The only way to determine which Capabilities an organization does or does not have is to perform a Capability Assessment, in which the Capabilities of a Best Practice (as outlined in the Improvement Directory) are assessed for their factual existence as demonstrated by evidence (i.e. outcomes, key performance indicators, metrics). The High Level Assessment (151 questions) is a sociological exercise, i.e. a survey of people's perceptions. It results in an inference, which is not the same thing as an assessment of fact or evidence. Indeed, it is an inference of a very imprecise nature based on reported perceptions regarding general things (best practices, not the constituent capabilities of best practices). It is a very general yet educated guess, but a guess all the same. This is extremely different from the more detailed or "comprehensive" assessment of Capabilities to which the Knowledge Foundation alludes.
Bridging the Gap
Craig writes that “The biggest key benefit that OPM3’s promoters are touting is that the tool will illuminate the link between projects and business strategy and bridge that gap.” He goes on: “This is the crux of the theory of organizational project management and where the OPM3 falls down the most (…) nowhere in the product is there a discussion on how it will link better project management to the business strategies of the organization.” Not true. OPM3 addresses the link between projects and business strategy in detail. Actually, OPM3 concentrates on two main things: 1) choosing the right projects, and 2) delivering those projects successfully. The first of these two main things has most to do with portfolio management, whereas the second of these two main things has most to do with program and project management. I draw your attention to “portfolio management” best practice 5660: the organization manages the value of the portfolio. Best practice 5660 is a decomposition of the formula for Black-Scholes option pricing. Such techniques are the very stuff of managing the strategy-project link. This best practice is an example of content pertaining to portfolio management in OPM3 that does not derive from a simple extrapolation of the PMBOK's 39 processes. While this one emphasizes the strategic selection of projects, others emphasize other aspects of the strategy-project link, including best practices1640, 1650, and 2170. See http://www.pmforum.org/library/papers/SchlichteronPPM.pdf.
Widely Endorsed
Finally, Craig writes that “PMI’s original vision for the product was to create a widely and enthusiastically endorsed maturity model that will become the standard in project management (… but) just because you say something is a standard, does not mean it is.” Actually, if you are an accredited standards developer like PMI, then something is a “PMI standard” precisely because PMI says it is (provided PMI judges that its requirements for standards development have been satisfied). This is an important distinction. PMI has requirements for the development of its standards that include open participation and consensus building and other things. Many of these requirements come from the American National Standards Institute (ANSI), which is a counterpart (member) to the International Standards Organization (ISO). The PMBOK Guide, for example, is an ANSI standard and was integrated into the OPM3. Because OPM3 is new, it must continue to benefit from close scrutiny and improvement until it becomes widely and enthusiastically endorsed, approved by both standards-setting bodies and new users as a de facto standard.
This natural evolution of things would be required even though OPM3 already incorporates an ANSI standard, even though requirements for OPM3’s development were validated by over 30,000 people, even though OPM3 was developed by over 800 volunteers in 35 countries over the course of half a decade, even though at its core OPM3 is based on the principles of standardization and measurement and control and continuous improvement that have been de facto standards within industry and government since the days of Shewart and Demming decades ago, even though it incorporates portfolio management principles borrowed from de facto standards in portfolio management endorsed within the financial management community today, and even though OPM3 was beta tested by Canon Europe, Ceridian, The Dow Chemical Company (TDCC), Eastern Exterior all Systems, EDS Ireland Solution Centre, Info-Tech Research Group Inc., Instituto de Informatica e Estatistica da Solidariedade (IIES), Lockheed Martin Aeronautics, Public Works and Government Services Canada (PPD PMO), SSM Health Businesses Information Center, TELUS, and Wachovia Corporation.
You may contact me at jschlichter@opmexperts.com.
--------------------------------------------------------------------------------
[1] While the 151 questions in the software on the CD are not necessary for a Capability Assessment, the CD does contain some other stuff that is essential for anyone who wishes to use the OPM3 for a Capability Assessment.
[2] The “maturity score” expressed as a percentage resulting from the “High Level Assessment” using 151 Yes/No questions simply denotes the number of those 151 questions that were answered in the positive.