Abstract

This paper reviews and interprets the use of evidence based policy making (EBPM) by the New Labour government since 1997. New Labour has used EBPM as a key part of its strategy of modernisation. However, the paper reveals that this development needs to be interpreted with some caution. Firstly, because EBPM does not represent a return to a technocratic form of government: issues such as power, politics and people remain critical. Secondly, EBPM has different meanings across policy domains and within social science: it is a contested concept. The paper concludes with an assessment of EBPM, arguing that evaluative research undertaken with an understanding of political ideas, institutions and contexts provides a richer basis to inform policy and practice. What counts may be what works; but understanding and identifying what works is not a simple technocratic task, but a tellingly reflexive one.

Introduction

We will be a radical government. New Labour is a party of ideas and ideals but not of outdated ideology. What counts is what works. The objectives are radical. The means will be modern. Britain will be better with new Labour.

Tony Blair, Labour Party Manifesto for the 1997 General Election

This Government has given a clear commitment that we will be guided not by dogma but by an open-minded approach to understanding what works and why. This is central to our agenda for modernising government: using information and knowledge much more effectively and creatively at the heart of policy-making and policy delivery.

David Blunkett, Speech to the ESRC (2 February 2002)

‘What counts is what works’ and ‘what works and why’ have been important elements of New Labour’s approach to government and in particular to the implementation of large social and economic development programmes. The 1997 Labour Party Manifesto and the speech by David Blunkett closely associate an agenda of evidence based policy making (EBPM), ‘understanding what works and why’, with a central element of New Labour’s political strategy: namely the modernisation of government and the wider apparatus of the state. The notion that policy-making should be ‘evidence-based’ rather than based on unsupported opinion is difficult to refute. However, it also poses a considerable number of normative questions, for instance, how should evidence be collected, what evidence should be used and how should that evidence be used. More broadly, EBPM, superficially at least, appears to deny a role for traditional factors in policy-making, namely, power, people and politics (Parsons, 2002). For some, this has been interpreted as a return to a technocratic style of policy-making (Sanderson, 2002a).

There is no single unifying account of EBPM. It is used in different ways across the policy and academic worlds.  The former Deputy Chief Social Researcher, Philip Davies, has argued that a ‘problem for evidence-based policy is the uncertainty of social scientific knowledge, and the different status of different fields of knowledge’ (Davies, 2004, p. 4). Mulgan suggests different fields of social and economic knowledge run along a continuum from fields of knowledge that are well established and almost like normal science to those where knowledge is inherently novel, such as global governance, regulation of biotechnology and e-government (Mulgan, 2003; cited in Davies, 2004, p. 4). This is reflected in the rapid growth of literature across academic disciplines on EBPM (see for example Davies et al. 2000, Sanderson, 2002a and 2002b, Campbell, 2002, Packwood, 2002) and within the evaluation (Pawson, 2002), statistics (Smith, 1996) and public finance (Nutley and Davies, 1999) communities. These literatures reflect the traditions of different areas of social and economic policy making but also the different epistemological, ontological and methodological approaches to social and economic research. The different theoretical and methodological underpinnings of EBPM is explored by this paper and in particular how specific forms of ‘evidence’ have gained increasing influence.

Despite these different conceptions of EBPM it is also apparent that the term is used with varying degrees of rigour. This is illustrated by an announcement by Alun Michael (as Rural Affairs Minister) in 2002 that the ‘decision on hunting with dogs [was] to be evidence-based’ (DEFRA, 2002). In this case, although the policy-making process would undoubtedly draw on scientific evidence, this decision is one that would eventually be taken by parliament. Such a decision undoubtedly drew on issues of political allegiance and values as well as the evidence provided by the Minister. This example highlights most starkly, as previous similar ministerial announcements over the health risks of ecstasy, BSE/CJD, foot-and-mouth disease, GM foods and crops, and most recently climate change, that the use of ‘evidence’ must be understood both in relation to issues of power and politics but also the factors that condition policy making, namely contingency and expediency.

The broad theme of this paper is that EBPM should not solely be understood within a perspective of instrumental rationality (Sanderson, 2002a; 2002b) which stresses the managerialist and technocratic understanding of the state. Instead it should be understood through different theoretical perspectives of governance both in relation to the analysis of the state as a co-ordinator of actors and interests (Flinders, 2002) but also in terms of the broader societal co-ordination of a complex array of organisations (Kooiman, 2000). That is, EBPM needs to be explained alongside other conceptualisations of public policy making.

The paper is divided into three sections. The first considers the rise of EBPM under New Labour, its different guises and meanings, and its role within a wider agenda of modernisation. The second section reviews different theoretical accounts of EBPM. The third concluding section reflects on the prospects for EBPM, in terms of seeking an ‘evidence informed society’.

The rise of evidence based policy making

EBPM was seen by New Labour as a central element of its plans for the modernisation of government. For example, the Modernising Government White Paper (1999) states that:

‘… policy decisions should be based on sound evidence.  The raw ingredient of evidence is information. Good quality policy making depends on high quality information, derived from a variety of sources – expert knowledge; existing domestic and international research; existing statistics; stakeholder consultation; evaluation of previous policies …’ (Cabinet Office, 1999, p. 31).

Moreover, the rationale for EBPM is set out as:

‘ … evidence-based policy helps people make well informed decisions about policy, programmes and projects by putting the best available evidence from research at the heart of policy development and implementation’ (Davies, 2000, p. 366; cited in Wyatt, 2002, p. 25).

The White Paper acknowledges evidence can take different forms, but that there are other influences on policy and that these include: experience, expertise and the judgement of policy officials and Ministers; values and ideology; available resources; habits and tradition; lobbyists, pressure groups and the media; and the pragmatics and contingencies of everyday political life (Davies, 2004, p. 21-23).  Davies also highlights that ‘the driving force for evidence in government tends to be the type of question being asked, rather than any particular research methods or design’ (Davies, 2004, p. 21). However, research-based evidence is also generated in different ways reflecting both the inherent nature of different policies, the differing epistemological and ontological bases of different professions and different approaches of social science disciplines.

The first term of the New Labour government included the launch of a series of major social and economic programmes. These included the New Deal for Communities, the employment New Deals, On-Track, Sure Start and the Children’s Fund. These programmes are subject to large scale evaluations. These programmes had a strong focus on delivering the government’s strategy for social inclusion and neighbourhood renewal (Social Exclusion Unit, 1998). These interventions have also been interpreted (Annesley and Gamble, 2003) as reflecting a new approach to the delivery of welfare policies in the United Kingdom and a shift away from the provision of a universal system of welfare to one which is closely targeted on specific groups and areas, and on those of working age who are not active in the labour market. Moreover, the rationale for these interventions is based on addressing the ‘wicked issues’ of policy making and seeking to join-up policies and agencies to address deep-seated social and economic problems. Findings from one of these long-term evaluations are reported by Lawless (2007) elsewhere is this issue.

The New Labour government has placed considerable emphasis on EBPM. However, its attitude to EBPM has also evolved from seeing it as an integral component of flagship social and welfare policies (such as NDC) to one where it is used in a more focused and selective way with greater attention being given to quantitative and economic analysis (DTI, 2007; HM Treasury, 1997; Performance and Innovation Unit, 2000). This is probably for a variety of reasons: a shift in the emphasis from policy learning to policy delivery; a concern that what matters most is hard quantitative data; and more generally a shift in the style of government, away from experimentation and towards delivery.

The following section sets out different theoretical accounts of EBPM, beginning with the notion that EBPM is part of a technocratic project of instrumental rationality and concerned primarily with identifying information on ‘what works’ rather than developing an understanding of ‘what works, for whom, in what circumstances, and why’ (Pawson and Tilley, 1997).

Theoretical accounts of evidence based policy making

Young et al. (2002) set out a descriptive and historical account of EBPM. They identify the following five models.  The first is a knowledge-driven one whereby the expert is ‘on-top’. This model reflects most closely the technocratic interpretations of policy development. Examples include 1950s and 1960s technology policy where scientists were able to shape and lead major programmes of investment, in fields such as nuclear energy and supersonic civil aviation. The second approach is the problem-solving model whereby the expert is ‘on-tap’. This model reflects government departments commissioning approach to research. The third approach is the interactive model whereby academics cross between academic and policy communities. This is seen to resemble some accounts of United States EBPM whereby, in comparison to the United Kingdom, academics more often spend time in government departments, but also the caricature of expert advisory committees in the United Kingdom as ‘good old boys sitting around talking turkey’ or GOBSATT (Davies, 2000; cited in Davies, 2004, p. 10). A contemporary version of this is expert led policy review. The fourth approach is the political/tactical model whereby policy is the outcome of a political process and social science is politicised. The final approach to EBPM is the enlightenment model whereby research stands aloof from policy making and that policy is evidence informed, rather than evidence based (Nutley et al, 2003). These accounts of EBPM are essentially static heuristic descriptions and they sit somewhere along the continuum set out by (Solesbury, 2001, p.9) as: ‘emphasising the role of power and authority at the expense of knowledge and expertise in public affairs seems cynical [the political model]; emphasising the latter at the expense of the former seems naïve [the enlightenment model]’ (Solesbury, 2001, p. 9).

Examples of each of these models can probably be found to some degree across government in the United Kingdom. However, this would be to deny the existence of a dominant model of New Labour’s approach to EBPM. Sanderson (2002a, p.5) argues that it is a project of instrumental rationality in which, citing Schwandt (1997, p. 74), ‘… policy makers seek to manage economic and social affairs “rationally” in an apolitical, scientised manner such that social policy is more or less an exercise in social technology’. Furthermore, Sanderson argues that this model is predicated on a series of assumptions relating to: ‘the nature of knowledge and evidence; the way in which social systems and policies work; the ways in which evaluation can provide the evidence needed; the basis upon which evaluation is applied in improving policy and practice’ (Sanderson, 2002a, p. 5). Dryzek argues that policy making by instrumental rationality is more akin to the ‘policy sciences of tyranny’ than to democracy (Dryzek, 1989, p. 98). Lasswell made this same point in the 1950s in his critique of technocratic policy making (Lasswell, 1951). Moreover, EBPM can be seen as a managerialist and mechanistic way of thinking about policy making.

Parsons (2002) makes a similar point to critique to Sanderson. Using the Schönian metaphor of the policy swamp (Schön, 1979), he argues that ‘knowledge for EBPM is a means of controlling the mess, and draining the swamp’ (Parsons, 2002, p. 45). Moreover, this explanation of EBPM marks a ‘return to the quest for a positivist yellow brick road … from which government can exercise strategic guidance’ (Parsons, 2002, p. 45). In the model of instrumental rationality, EBPM is closely tied to the politics of a strong central state, of centralisation, and of command and control. EBPM is therefore closely aligned to the modernisation agenda themes of audit, inspection and monitoring rather than to the themes of learning, improvement and involvement.

This conceptualisation of EBPM can be criticised from a range of perspectives. From a constructivist viewpoint ‘knowledge of the social world is socially constructed and culturally and historically contingent’ (Sanderson, 2002a, p. 6) in which policy delivery is seen as a ‘process of deliberation under conditions of multiple frames for the interpretation and evaluation of the world’ (Dryzek, 1990, quoted in van der Knapp, 1995 p. 202). Alternatively, under post-positivist analyses of policy making, greater emphasis should be given to factors such as policy networks, policy transfer networks and the way in which ideas can shape policy-making (Mazey, 2000).

Similarly, EBPM has also been interpreted as a part of reflexive policy making and practice. For example, Nutley et al. (2002) argue that much greater emphasis should be given to how evidence informs policy and how both policy makers and practitioners learn and reflect. Authors such as (Jessop, 2001) refer to this as ‘reflexive monitoring and dynamic learning’. Parsons, reflecting arguments on complexity and networks (Chapman, 2002; Bentley and Wilsdon, 2003) argue that the state should not be seen as a bureaucratic monolith, and that command and control are unsuited to the ‘unpredictable demands of contemporary organisational life’ (Parsons, 2002, p. 50). Instead, the challenge for Parsons, Mulgan and Jessop is that public institutions and governance regimes should be recreated as ‘open, porous and decentred systems which can thrive on diversity, adapt to radical innovation and still remain coherent to a purpose of progress’ (Parsons, 2002, p. 50). This reflects arguments that ‘traditional ways and means of governance are no longer adequate…traditional structures of authority and problem-solving have failed’ (Kooiman, 1993 pp. 249-50).

The emphasis placed on learning, on the role of networks and on the extent of reflexivity in the policy making process appears to ignore contexts and settings which may shape the operation of these processes. EBPM places too much emphasis on the role of causal processes in improving policy effectiveness and insufficient emphasis on the institutional and organisational context in which policies are made and implemented. In contrast new institutionalism (March and Olsen, 1989) would posit that informal values, norms, interests and ‘taken for granted beliefs’ shape policy making and provide an informal organisation in which policy making is played out (Lowndes, 1997). Sanderson argues that such an informal organisational setting provides the ‘informal rules in interpreting meaning and providing guidelines for action for organisational agents within a social network’ (Sanderson, 2002b, p. 67).

Evaluating New Labour: up to and beyond 2007

This paper has reviewed literature which highlights how EBPM varies across areas of government, reflecting past practice and its institutional setting, professional practice, as well as the different epistemological bases of different policy areas. Within economic and social policy, the paper argued that the dominant mode of EBPM is one of instrumental rationality. (Sanderson, 2002, p 19) argues that:

A focus on reflexive learning is required to resolve the paradox of late modern society: that while increasing complexity of social systems progressively undermines the notions of certainty in social knowledge it simultaneously raises the stakes in relation to rational guidance of those systems.

Although true it is also doubtful to what extent the reformist projects of the adaptive and reflexive state can be achieved.

There has also been a shift in the way evidence is captured and used by policy makers. Alongside the formulations of EBPM set out in this paper, the New Labour government has also given a renewed role to experts through the establishment of high level reviews. Examples of these include: the Barker Review on Planning, the Leitch Review on Skills, the Lyons Review on Local Government, the Gershon Review of Public Sector Efficiency; and the Stern Review on the Economics of Climate Change. The use of such reviews can be interpreted in different ways: they provide a mechanism through which wide ranging evidence can be collected and then formulated into policy recommendations; as a means of keeping evidence at arms length, as only some recommendations need to be taken on board; as a mechanism for legitimating a policy by giving it an expert’s authority; and possibly of bringing critical voices within a framework which can more easily be managed by government.

Despite shifts in the use of evidence by the UK government, there do appear to be a set of tangible changes in policy making at all levels. Firstly, evaluation has become a more widely accepted part of the policy making process, more frequently and knowledgably used by central government and local and regional agencies. For instance, evaluation designs have become more sophisticated with greater use of a range of data, including longitudinal elements as well as theoretically based approaches. Secondly, and perhaps reflecting the current point of the political cycle, there is less emphasis on ‘good practice’ and ‘what works’ guides and more on ‘evidence reviews’ and evidence assessments’. This perhaps reflects both an acceptance that what works in one place may not in another, but also that considerable evidence already exists and perhaps could be explored more effectively. Thirdly, there remains a perennial concern with outcomes and cost effectiveness.  However, as with value for money studies in the 1980s and 1990s these are only as useful as the quality of data and the clarity of assumptions made which guide them.

The critique of EBPM that it marks a return to technocratic government has prompted debates and challenges which stress the importance of learning from evidence and through practice. This appears to happen best when policy making and practice has sufficient reflective space, and not necessarily when expediency, contingency and exacting performance frameworks squeeze out opportunities for experimentation and innovation.

Finally, EBPM is only one component of the policy making process. Ideas, values, political strategies and previous practice are probably of greater significance. However, as this review has shown, evaluative research undertaken with an understanding of political ideas, institutions and contexts provides a richer basis on which to inform policy, and equally, practice. What counts may be what works; but understanding and identifying what works is not a simple technocratic task, but a tellingly reflexive one.