By Brian Head
There has been a growing literature on ‘wicked problems’, which are generally seen as complex, controversial, intractable and evolving. Both the nature of the ‘problems’ and the best ‘solutions’ are strongly contested by political parties and various stakeholders. The problems are ‘wicked’ in the sense that they are not easily tamed and resolved.
Some policy researchers are evidence-optimists, in the sense that they hope evidence-informed policy arguments can shape policy reforms in ways that generate clear improvements. While they recognise that political ideologies and partisan stakeholders can obstruct rational reforms, the evidence-optimists argue that using best-available evidence can lead to measurable social, economic and environmental benefits.
Historians are generally sceptical about the assumption that the story of policy change is about steady progress. There are too many instances where hard-won benefits have been overturned or reversed, and too many issues which have become gridlocked. Some writers have suggested that when the problems are complex and when the stakeholders hold divergent values, it becomes much less likely that effective problem-solving will be possible. This is the domain of enduring wicked problems. My new open-access book Wicked Problems in Public Policy (Palgrave) outlines the history of this concept and applies it to several major areas of policy conflict and crisis: https://link.springer.com/book/10.1007/978-3-030-94580-0
The concept of wicked problems emerged during the era of Lyndon B Johnson’s ‘Great Society’ programs in the late 1960s, when there were polarised views about how to address major problems of inequality, unemployment and racial discrimination in the big US cities. Debates raged between the rational planners, the incremental conservatives, and the leaders of protest movements. Could federal programs solve these problems? Federal money began to flow into city administrations to fund public works projects; federal legislation attempted to secure civil and political rights; and several programs were established to improve social security and welfare.
Amid the turmoil, city planners were developing schemes to improve the built environment and social researchers were developing proposals to improve the inter-connected issues of jobs, education, health and criminality. Academics at the University of California, Berkeley, joined in the social analysis and the policy debates. On the one hand, the public policy and management scholars, led by Wildavsky, argued that the program was poorly designed and poorly implemented because it presumed excellence in coordination and high levels of information. On the other hand, the urban design scholars, led by Rittel and Webber, argued that complex problems could only be advanced through dialogue and collaboration, taking account of the inherent differences in stakeholders’ values and perspectives. They coined the term ‘wicked problems’ to designate the underlying dilemmas of working with multiple stakeholders and the impossibility of reaching a ‘correct’ solution.
The social sciences have debated these issues for fifty years, without a clear outcome. Many of the historical case studies around contentious policy issues have demonstrated the difficulties of achieving meaningful reform on complex matters. And most of the literature about policy responses to crises – economic, political, public health, natural disasters, and so on – has repeatedly shown deficiencies of leadership, coordination, and contingency planning. Crises can play a double role in this story, both as the direct cause of harm and suffering, and as an opportunity to reset the policy directions for building a better future.
Much emphasis has been placed on the failures, but there are also examples of where genuine progress has been made on tough issues. Some are discussed in a recent symposium on the lessons from effective policy reform, edited by Brown and Dovers: https://socialsciences.org.au/publications/principles-of-effective-policy-reform/
The supply of evidence-informed and well-crafted policy ideas is never enough. But even when excellent proposals are available, there is no automatic translation into better decisions and better outcomes. This malaise has led to closer examination of how evidence and values intersect in complex policy arenas. Complex and contentious issues are inherently marked by differences in values, interests and perspectives. Many issues with long histories of policy conflict display those embedded differences of perspective – for example, debates on immigration and refugees; poverty and homelessness; recognition of indigenous First Nations’ rights; sustainable forms of economic development; tackling illicit drugs; and balancing equity and efficiency goals.
On such matters, value differences cannot be effectively sidelined or ‘de-politicised’ through appealing to evidence and data, or by referring the issues to a committee of technical experts. In many cases, governments become impatient and impose their own preferred solution, either to appease their supporter base (‘keeping promises’), or simply to close down the debate. However, in the long run, these underlying differences in values and interests need to be acknowledged, and more collaborative methods found to accommodate and respect diversity. These are genuine challenges for political leadership as much as for evidence and expertise.
There are no ‘correct’ answers to wicked problems. But some policy frameworks and stakeholder inclusion processes are better than others, both in terms of gaining wide support (legitimacy) and achieving progress toward desired goals (outcomes). Improvements are possible over time. Historical examples suggest that inclusive processes for considering the problems and debating possible reforms are more beneficial than ideological solutions imposed by government.
