Introduction
There has been much discussion on the forums about complex adaptive systems, agility, “all processes are design processes,” etc. This triggered a thought about Mary Poppendieck’s mentioned of the ability of agile methods to address “wicked problems.”
The concept of "wicked problems" in design was originally proposed by H. J. Rittel and M. M. Webber (1973, “Dilemmas in a General Theory of Planning”) in the context of social planning. They pointed out that in solving a wicked problem, the solution of one aspect may reveal another, more complex problem. Rittel and Webber suggested that the following rules define the form of a wicked problem:
So what’s the problem here? The answer is wicked problems possess attributes that make them difficult to manage in a “top down” manner. And what’s does this mean an agile project manager? First is to recognize that problem solving is opportunity driven not plan driven. The challenge then is to create the opportunities. In plan driven processes, plans by their nature create the road on which to drive. “Plan the work, work the plan,” is a euphemism. From PMBOK, inputs to the planning process include: historical information, policies, constraints, assumptions. These result in a “plan” for the project. But this plan represents the fictional promise of those who built the plan in the absence of any tangible evidence that the plan can work. Sure there is experience, past performance, even assurances from all parties that they can deliver on their promises. In the end though it’s still fiction. Truth only comes during the “driving” phase.
Criticism of the “Wicked Problems” Paradigm
The ten points listed above are not without their critics. First, “wicked problems” share many of the attributes of “ill defined problems.” Ill-defined problems have no clear goal in mind. Finding the solution requires further definition of what the problem is. These problems belong to a class of problems called “Learning by Design (LBD).” As well they belong to the “group knowledge” domain. This connection is not obvious at first but hopefully I can build this bridge without too much pedantic hand waving. The primary criticism of what’s to follow is likely to be – this is too soft for any real project management to deal with.
How Humans Approach Wicked Problems
The first attempt to remove the ambiguity present in wicked problems is usually to produce “artifacts” of the goal, the planned progress toward the plan, and activities that are performed along the way to the goal. The result is usually documents of some kind. These documents represent specifications, agreements, assessments, more plans. “Plan in depth” is a phrase I remember from my distant aerospace days. In this artifact oriented world most of the work is focused on the production and review of these documents. In the vernacular of Rummler-Brache (Improving Performance: How to Manage the White Space on the Organization Chart, 2nd Edition, Jossey-Bass, 1995) the failures appear when there is no means of supporting the messy, informal processes that are described by these documents.The first approach when encountering a wicked problem or project is to jump to our tried and true reductionist techniques:
The next approach is to apply some kind of methodology against the problem that may be non-traditional. This is where we find agile methods – either development or project management. Both these approaches (reductionist and agile) fail to recognize the fundamental difference between wicked problems and hard but ordinary problems. Wicked problems are primarily “issues” problems not “solutions” problems. They are “creation” problems rather than “assembly” problems. Solutions to wicked problem arise through intuitive processes rather than logical processes.
Here’s where the standard – read traditional, linear, structured, normative – approaches get in trouble.
"Some problems are so complex that you have to be highly intelligent and well informed just to be undecided about them – Laurence J. Peter"
It’s the decidability aspects of the problem that create the trouble. There has been some discussion in the forums about the differences between hardware and software. I contend the primary difference is the number of indeterminate states for software is larger by many orders of magnitude than hardware. This creates a “decidability” issue – “when is the software actually working?” This in turn leads to the wicked problem type of project. Which is tied to the determinacy and decidability attributes of the problem.
A Quick Side Bar into Post Normal Science and Other Topics
Traditional project management methods are based on scientific principles that would be considered “normal science,” but they lack any theoretical basis for this approach. These principles make use of linear step–wise refinement of the project management processes based on a planning–as–management paradigm. Plans made in this paradigm and adjusted by linear feedback methods cannot cope with the multiple interacting and continuously changing technology and market forces found in any modern software development project.
The success rate applying traditional methods to complex software development projects over the years has been underwhelming. This linear step–wise approach has its roots in the waterfall methods of the 1970’s. It was clear then and has become even cleared today that this approach to managing software projects is inappropriate in many domains. What is not answered in the project management literature is the question – is there an underlying theory of project management? A secondary question is – can a theory be constructed that is consistent with adaptive system and agile processes currently in use in manufacturing, science, economics, biology, and ecology?
The normative advice provided by bodies of knowledge – planning, execution, and control – forms a closed loop linear system. This advice is usually based on rules that specify which choices will maximize benefits to the participants. Normative theory suggests a project is a series of linearly dependent activities. (Be careful here to understand the difference between linear and non-linear. Linear can have multipliers on the variable, but not in the exponent.) In practice software project management is a set of multiply interacting interdependent activities behaving in a non–linear and adaptive manner. Complex adaptive systems (CAS) are one way of looking at project management. Adaptive control systems offer another model without the complex and intractable mathematics of CAS.
Wicked problems, post-normal science and agile methods connections
The term “Post–Normal” was coined by Funtowicz and Ravetz. A simple definition is… “(g)oing beyond the traditional assumptions that science is both certain and value–free, it makes system certainties and decision stakes the essential elements of its analysis.” It distinguishes between “applied sciences” where both dimensions are low, “professional consultancy” where one of the dimensions is salient, and “post–normal science” where at least one dimension is extreme.
These concepts form the basis of many of the “agile” processes in management and development of software systems:
Summary
There are four strategies for coping with wicked problems. Notice the term “coping” rather than solving.
In the end the three (3) approaches to wicked problems mentioned at the top of this editorial have now come full circle, but now with four elements. So what does this mean for us agile project management types? First is to understand when we have a “wicked problem,” and when we don’t. When we do, resist applying the reductionist and normative approaches in search of a solution. Finally the tenants of agile match closely with those of “wicked problems.” More connections and discussion is likely to prove useful.
References
This is a complex topic and like most complex topics is subject to over simplification, improper reuse of both the ideas and solutions. I suggest that anyone interested in the “real” problems associated with “wicked problems” take a look at the resources and references below. These are a small sample though and more research is needed: