This article is written by Jonathan Shepherd, surgeon and criminologist, based at Cardiff University’s Crime and Security Research Institute.

Throughout the 20th century, randomised trials were increasingly used to test the effectiveness of surgical and medical interventions in healthcare. Since then, they’ve become common practice in other areas, such as education, policing and local government.

The spread of this rigorous evaluation method has been accompanied by a growing volume of systematic and other reviews of evidence on intervention effectiveness and of guidance produced by thousands of global organisations.

The result has been a tsunami of evidence and guidance.

Not surprisingly, a UK government minister said recently that he is “overwhelmed by evidence”. The special adviser of the UK Cabinet Minister for Education reflected that, “I am quite cynical of evidence presented to me as everyone has ‘evidence’ to back themselves up”. Commenting on  the UK’s pioneering What Works Network, which comprises sector-specific centres and a co-ordinating council, a senior policymaker observed that: “Each centre has different evidence standards and resources. Expecting policymakers to get to grips with each one is unreasonable.”

These were the most striking, and surprising findings in my 2020 report, Evidence and Guidance for Better Public Services: Making the Most of the Evidence Ecosystem.

Public servants need to be highly selective in the evidence and guidance they seek and use. Without quality assurance marks on trials, evidence reviews and guidance reliability remains doubtful

The reasons behind them however are not hard to work out. Today’s public servants work in an age where all guidance is expected (or assumed) to be “evidence-based”. Yet, because this evidence is often not compliant with defined quality standards, evidence that makes it into guidance can be very quickly obscured by warnings and qualifications.

Some intervention trials and systematic reviews, but relatively little guidance, conform to internationally recognised standards. Most do not. Worse still, this avalanche of evidence acts as a deterrent to using or even seeking guidance altogether. This means that genuinely valuable guidance is overlooked, to the overall detriment of decision-making.

Public servants, therefore, need to be highly selective in the evidence and guidance they seek and use. Without quality assurance marks on trials, evidence reviews and guidance reliability remains doubtful. Since guidance has the potential to generate harm as well as good, these safeguards are crucially important.

Helpfully, international quality standards exist. An accreditation scheme for organisations which publish guidance is also available, though currently only in health and social care in the UK. Crucially, these standards should be applied and quality marks displayed prominently on all compliant trial reports, evidence reviews and guidance. Public servants should look for signs that all trials comply with national as well as international research requirements: for example, those in the National Health Service Research Governance Framework. Trials should be reported and interpreted according to the Consolidated Standards of Reporting Trials (CONSORT).

Systematic reviews of evidence, meanwhile, should include assurances that they have been carried out and reported according to Cochrane or Campbell collaboration standards.

Without standardisation, progress towards this objective will slow and may even be reversed as decision makers are overwhelmed by unregulated, non-standardised evidence and guidance

The Appraisal of Guidelines for Research and Evaluation (AGREE) Instrument, AGREE II, is a valid and reliable, internationally recognised standard for the production of practice guidelines. In the UK, the National Institute for Health and Care Excellence (NICE) accredits guidance producers in the health and care sectors according to this standard, and their guidance bears the NICE Blue Iris quality mark. This Accreditation Programme for guidance producers should be extended, and widely advertised across the public sector.

As countries recover from the Covid-19 pandemic and its economic impacts, public services need to be as effective and efficient as possible. The way to achieve this is a fully functioning, global Evidence Ecosystem in which evidence production, synthesis, and translation into reliable guidance for policymakers and practitioners is standardised.

But without standardisation, progress towards this objective will slow and may even be reversed as decision-makers are overwhelmed by unregulated, non-standardised evidence and guidance. What’s at stake here is the public good, which scientific evaluation continues to give us the opportunity to deliver. — Jonathan Shepherd

(Picture credit: Unsplash)

Make sure to share your own thoughts with the author by leaving a comment below