As a society we face some big challenges. Take your pick from a list that includes climate change, the Fourth Industrial Revolution, an ageing population, urbanisation and the growth of cities, and the rise of populism.

If ever there was a need for innovation in policy-making it is now.

Since the late 1990s one key innovation in policy-making that has gathered widespread support has been the move to Evidence-Based Policy (EBP). In its ‘traditional’ form EBP gives precedence to experimental evaluation designs and the use of systematic reviews to interrogate the evidence base. It’s a model that has drawn heavily on the concept of evidence-based medicine, which brought about a paradigm shift in health. As the author Ben Goldacre notes:

“In the nineteenth century, as the public-health doctor Muir Gray has said, we made great advances through the provision of clean, clear water; in the twenty-first century we will make the same advances through clean, clear information. Systematic reviews are one of the great ideas of modern thought. They should be celebrated”. (Goldacre 2009, Bad science, London: Fourth Estate p. 98)

In the UK successive governments of different political persuasions have invested in EBP, in part by building a network of What Works Centres. Despite occasional skepticism about experts, the network has expanded considerably under successive Conservative and Conservative-led governments. However, evidence-based policy has yet to deliver the step-change across social and economic policy that was seen in medicine and many medical practices are not evidence-based. For example, although the evidence is clear, childhood vaccination rates are dropping and although its use is dropping in primary care, prescribing levels of antibiotics remain too high and are rising in secondary care. It seems that the challenge of knowledge mobilisation is considerable and implementation of evidence-based policies will often depend on changing practice in front-line services, which in turn requires effective behaviour change interventions where the importance of human factors in implementation is understood.

Systematic Reviews are only one advance in the growth of ‘clean, clear information’. More recently, the debate on better policy-making has turned to the role of technologies such as AI and the potential of Big Data. Will these come together in applications such as ‘Seeing Rooms’ and help policy-makers understand complex systems and argue with evidence? In the same blog in which he appealed for ‘weirdos and misfits’ to work in government, the Prime Minister’s Special Adviser Dominic Cummings argues:

“There is very powerful feedback between: a) creating dynamic tools to see complex systems deeper (to see inside, see across time, and see across possibilities), thus making it easier to work with reliable knowledge and interactive quantitative models, semi-automating error-correction etc, and b) the potential for big improvements in the performance of political and government decision-making.”

One interpretation of Cumming’s now infamous blog is that he is attempting to engineer a change in the policy paradigm. Peter Hall, probably the leading thinker on the political economy of policy making, was inspired by the concept of scientific paradigms developed by Kuhn and describes a policy paradigm as “ an overarching framework of ideas that structures policy-making in a particular field”. For Hall the policy paradigm specifies “not only the goals of policy and the kind of instruments that can be used to attain them, but also the very nature of the problems they are meant to be addressing”. One criticism that might be made of Cumming’s approach is that, while he has a clear idea of how science and technology might be used to re-shape the policy-making process, a shift of policy paradigm is more likely to be driven by political than scientific considerations. Hall, who developed the concept of the policy paradigm, suggests that it is the group that can successfully redefine the problems that need to be addressed that defines the new policy paradigm. So, although the greater use of science and technology in policy-making is to be welcomed and will no doubt have a significant impact, it will be whoever, wins the debate about the defining the problems of our age who will shape the paradigm and these are the types of questions best answered by people, not algorithms, a view that seems to be shared by many of the scientists who were mentioned by Cummings in his blog.

This being the case, perhaps Cummings is drawing on the wrong model of innovation in his bid to develop a new policy paradigm. In the post-industrial, information economy, new models of innovation have started to break down the distinction between technologically driven and people-driven innovation. For example, the concept of Open Innovation, particularly recent manifestations, emphasises engagement between (i) industry, (ii) government, (iii) universities and (iv) communities and users to solve societal challenges sustainably and profitably. Open Innovation 2.0 inverts traditional models of technologically-led innovation so that innovation is driven by the creation of ‘ecosystems’ made up of a mixed economy of diverse actors who align their goals and collaborate to co-create ‘shared value’. Interestingly, instead of the user or citizen being seen as a research object and innovation being done to the citizen, in these models of innovation citizens and the people who use services are the drivers of innovation and work with government and services to co-create innovative solutions to pressing social challenges. This only happens when there are high levels of trust between collaborators and conviction in a shared vision. Open data and data science have a role to play, but technology is a facilitator of co-created solutions to social problems, not the driver of innovation.

I and my colleague, Professor Kevin Albertson argue that the key factors to promote genuine innovation include:

• Developing innovative ecosystems with deep collaboration between industry, government, universities, and communities and users.

• A collaborative approach where different partners work together in pursuit of shared value.

• A co-created and personalised approach both at the system level in terms of service design and delivery.

• A system which fosters localism in order to foster innovation.

• Greater investment in a broader understanding of evidence including prototyping and rapid experimentation accompanied by a more benign attitude to risk and failure to allow new approaches to be road-tested more quickly.

One implication of this approach to innovation is that disciplines across Universities have a role to play. Those working in the social sciences and humanities have as much to offer as the computer scientists and analysts.

Chris Fox is Professor of Evaluation and Policy Analysis at Manchester Metropolitan University, where he is also Director of the Policy Evaluation and Research Unit and co-founder of Metropolis a university thinktank helping to embed researchers in policy and practice.