At first glance, questioning whether governments should craft policy based on evidence seems senseless. Why wouldn’t we want programs to stand up to rigorous testing before we spend millions on them? Wouldn’t passing legislation backed by irrefutable data drastically improve citizens’ trust in government? Isn’t this just the norm?
For proponents of “evidence-based policymaking” (EBPM), the case is simple. “The dirty secret of almost all government is that we don’t know whether we’re doing the right thing,” said David Halpern, founder of the UK’s Behavioural Insights Team and one of the leading advocates for evidence in policymaking. “Whether what we’re doing is helping the patient or not helping the patient, or even killing him. People are often running blind, overconfident, and dispensing massive sums of money.”
“You could really improve people’s lives”
But strange as it may seem to people outside government, the emphasis on applying scientific standards of proof to policymaking is a recent and disputed phenomenon. Over the past 20 years, it has become a movement across the developed world, accelerated by the advent of behavioural insights and of big data, both of which promise to let policymakers measure their impact more precisely than ever before.
This movement is working towards a rationalist dream: that hard evidence can remove the partisan wrangling from policymaking and turn it into a scientific process, guided by numbers, run like a government innovation lab, and devoted not to political ideologies but to the simple question, what works?
The idea is popular. In the past two years, more than 100 US cities have pledged to use evidence and data to improve their policymaking under that slogan “what works”. In the UK, the What Works Centres, first conceived in the 1990s, are now responsible for billions in funding decisions.
But this movement is not without its critics. “EBPM is kind of a political slogan,” said Paul Cairney, a professor of politics and public policy at the University of Stirling in Scotland. “The biggest misconception is that evidence could ever be used at the heart of policymaking.”
Cairney, author of The Politics of Evidence-Based Policymaking, believes that EBPM is untenable. In his view, “there’s just too much evidence out there for anyone to consider” and policymakers consequently resort to shortcuts, basing things on one piece of research or one consultant’s advice, which negates the whole point. Ultimately, he believes that, in the pressurised world of politics, the standard set by EPBM is impossible.
What exactly is evidence-based policy?
For a policy to count as evidence-based, it must be evaluated using an empirical, scientific approach – ideally, a randomised controlled trial (RCT). Using a RCT, researchers can determine whether a target population fares better or worse under a specific social intervention than a control group does without it. If it does, the program can be said to work. It can then be rolled out to more people; if not, the RCT can help identify shortcomings.
The aim is to use unbiased reasoning to guide social interventions, and spend public funds more effectively. In an era where the truth seems dispensable to some politicians, EBPM champions the importance of getting the facts right.
In the UK, the concept was pioneered by former Prime Minister Tony Blair in the late ‘90s when the Labour party ran on the platform “what matters is what works” – the pragmatic, rational answer to ideology-driven politics. The goal was simple: to free policymaking from the everyday pressures of politics.
“We’re going to be funding a lot of things that aren’t effective”
Blair’s administration laid the foundation for the What Works Centres, 10 sector-specific bodies dedicated to meticulously evaluating policies in their area of expertise. Altogether, the What Works Network is responsible for £200 billion ($280 billion) of decision-making. It recently released a report outlining solutions that came out of its five years of experiments across sectors.
In the West Midlands, for example, police were struggling with how to deter people from unsafe driving. The language in the letters they sent to drivers caught speeding was too complex – as a result, offenders did not pay their fines and ended up in court.
“We started showing a picture of the aftermath of a car accident, where flowers and a teddy had been left on a lamp post, alongside the text: ‘Last year, 700 children were killed in this area because of speeding drivers’. It demonstrated that our motivation is good: it’s not about collecting fines; it’s about saving lives. And that’s when we saw the response,” said Alex Murray, Assistant Chief Constable for the West Midlands Police.
Drivers who received the revised letter paid fines 20% more quickly, and as a result, went to court 41% less – which saves the county about £1.5 million ($2.1 million) in court fees per year. The behavioural science-based approach has also cut reoffending by 20%.
These kinds of “nudges” have exploded in popularity all over the world since they were pioneered by the UK’s so-called Nudge Unit, founded in 2010. But the man who set it up, David Halpern, believes that the truly revolutionary aspect of its work is not the nudges themselves, but the commitment to rigorously testing their efficacy.
His unit would habitually create not one nudge, but several different messages with variants on the same idea. These would then all be tested against each other to see which was most effective. That one message would then be refined into more variants, which would again be tested. Vary, test, learn and repeat: the experimental method in action.
“Decisions should be made by people who actually know what’s going on”
“The idea you have to get people to understand is that just sort of muddling along, as we have done in the past, is not good enough,” said David Anderson, Director of EBPM at the Laura and John Arnold Foundation. The foundation funds RCTs in a wide range of policy areas, including the environment, education, and criminal justice, and works to advance the use of policies with a strong evidence base in the US.
“EBPM has demonstrated that by using rigorous evaluations, you can identify programs and strategies that lead to important outcomes, and could really improve people’s lives,” said Anderson. “The challenge is getting people to recognise that this is something they need to prioritise. If we don’t, we’re going to be funding a lot of things with good intentions that aren’t going to be effective.”
EBPM may be the gold standard for crafting policy – the question is how that gold standard copes with messy reality.
The replication problem
One of the biggest criticisms of EBPM is that a great deal of time and money can be spent testing a solution in one environment with a specific set of variables – then the program simply does not work in a different setting or with another demographic.
This was true of the Nurse-Family Partnership, a home-visiting program in the US that pairs nurses with pregnant women in low-income communities to teach new mothers about pregnancy and child-rearing.
The Nurse-Family Partnership was exceptionally successful in a California-based RCT: 92% of babies were born full-term, and 91% of newborns were vaccinated within 24 months. However, in other parts of the US – and in the UK – the program had an insignificant effect on the well-being of newborns.
“You can’t just assume that if a program was evaluated in one place in a modest-sized sample then it can work again somewhere else. You need replication trials or large, multi-site trials to begin with, where the results are consistent across different sites, demographics and job markets. Replication is a key piece to EBPM,” said Anderson.
Pressure on frontline workers
Because of EBPM’s emphasis on testing, some frontline service workers – who are already overstretched and under-resourced – are concerned that EBPM will be used to rationalise budget-cutting.
“Evidence-based policy doesn’t seem to be as much of a priority as it has been in the past”
Critics like Jerry Muller of the Catholic University of America argue that EBPM gives too much power to detached researchers and policymakers, leaving on-the-ground workers’ expertise and experience out of the decision-making process. Muller believes that this kind of over-reliance on data leads public servants to chase metrics that fail to reflect reality.
Discussing his book, The Tyranny of Metrics, with Apolitical, he cited a famous example from the UK: “When the National Health Service decided that a major problem was that people were having to wait too long to be admitted to emergency wards, they declared that hospitals would be evaluated based on to what extent patients were admitted within four hours.”
“Some hospitals responded by having the ambulances with patients circle around the hospital until they could be admitted within the four-hour window. People in their homes were waiting for ambulances to pick them up, while they were circling around the hospital in order to help it meet this metric. There are infinite varieties of gaming of that sort that occur.”
Muller added that: “Someone who is very far away from the actual practice – some civil servant whose knowledge of a particular practice is not extensive – is going to try to monitor and come up with a set of criteria to measure. Decisions should be made by people who actually know what’s going on.”
Everything qualifies
EBPM has become a buzzword, explains Anderson, with some politicians and public servants attributing it to any policy with some form of study backing it. The crux of EBPM is rigorous, unbiased research, typically in the form of an RCT. It must show considerable effects sustained over time.
“What often happens is there’s a downward pressure on the evidence standard to the point where basically just everything qualifies, and what you end up doing is lumping in programs that really do have strong evidence and improve people’s lives with less effective programs,” said Anderson. “We need to make sure the definition doesn’t get so watered down that it could be virtually anything.”
Future policy
While EBPM evaluations can measure the effectiveness of current programs, they do little to incorporate future policy trends.
If researchers were to evaluate a customer service model, for example, they may look at data points such as wait time or how many agents are needed to resolve an issue. Based on this information, policymakers would make changes to the current delivery model – without incorporating modelling for what citizens may want in the future. In this case, policymakers may be wasting time and money improving a service that’s already out of date.
Some policymakers have called for a shift from EBPM to predictive analytics, which uses data to identify patterns, helping governments to anticipate behaviours and demand. Others endorse greater incorporation of citizens’ opinions, without which research scientists may miss out on critical insights.
New administrations
Policymaking is usually far removed from the controlled environment of a lab. The people who craft policy are influenced by a wide range of powerful actors and lobbyists; rules and norms; party lines and ideological beliefs. A change in administration can shift everything onto a different track.
For example, in the US, former president Barack Obama was a big proponent of evidenced policymaking. He backed six major EBPM social initiatives, one of which was the Teen Pregnancy Prevention Program (TPPP), which replicated model programs across the country.
This is particularly significant for teen pregnancy prevention initiatives; the field is known for producing “boutique” models because programs tend to only be successful in one place and time. The success rate of most pregnancy prevention programs is 10-20%; the TPPP’s was 40%.
Apolitical
>> This piece was originally published on Apolitical, a global platform for public servants. It publishes authoritative writing on public policy and has a particular specialism in government innovation.
>> Denne artikel er oprindeligt udgivet på Apolitical, en global platform for offentligt ansatte. Apolitical skriver kompetent om offentlig politik og er særligt specialiseret inden for offentlig innovation.
Then, President Trump was elected and the political winds changed. “It doesn’t seem to be as much of a priority at senior levels of the administration as it has been in the past,” explained Anderson.
What makes an evidence-based approach vulnerable to political change is that governments don’t want to be seen to be wasting money on so-called experiments. Media headlines along the lines of “Government wastes millions on a program ultimately found to be ineffective” have the power to derail political careers, despite the fact that rolling out an untested program could very well waste billions in public money.
Technocracy and politics
Public policy researcher Jenny M. Lewis called EBPM “nothing more than a technocratic wish in a political world”. It appeals to our desire to rationalise policymaking and turn the business of government into a matter of endless refining, with ever greater knowledge and deeper expertise gradually wearing social problems down to nothing.
The ultimate limitation of that philosophy is that our societies have not agreed on what they are trying to achieve, nor on what they are willing to pay for it. It would be hard to find anyone who opposes, say, better nutrition for disadvantaged children. But as soon as you try to say whose responsibility the problem is – whether parents, schools, nutritionists or urban planners trying to break up “food deserts” – and then try to allocate the limited available funds to one or the other, you leave the realm of problem-solving and enter into a struggle over values.
A society is a system so complex that at times it borders on chaos, and it is a noble and necessary endeavour to tame some of that chaos with reason. Our increasing understanding of the forces at work among human communities is an echo of how the scientific revolution allowed humanity to explain the natural world.
But society is not nature. The apparently simple question – what works? – disguises that what policymakers are trying to achieve remains contested, and probably always will. So those of us interested in better policy and better societies would do well not to malign politics too harshly: after all, great leaps of progress are often made not through rational refinement but through feeling, persuasion and a vision that a society’s aims should be different. To put it bluntly, they are often made by politicians.
