If you are in the UK, you may know that the TV drama, Mr Bates vs. the Post Office, telecast by ITV during a slow week at the start of the year, has successfully managed to achieve what years of assiduous journalistic reportage, from Computer Weekly to Private Eye, could not: bring the Horizon/ Post Office scandal back on the agenda, leading to the British PM proposing legislation to quash over 700 wrongful convictions.
Now legal experts and judges are warning us (FT link; may require registration) about parliamentary interference in courts. Separation of powers among the legislature, the executive, and the judiciary is a fundamental building block of western liberal democracies. This exoneration-by-legislation muddies the waters and sends a poor signal although some are at pains to say this case is “an exception”.
They are right to raise the issue but this is not really just about the courts wrangling with the Parliament. The technical issues in the Fujitsu-developed Horizon software were many and are comprehensively described here by UCL’s information security research team. Also in the mix is a legal question: the evidential presumption that computers are operating correctly. David Allen Green’s whole post on the matter is worth reading.
This development in the Horizon/ Post Office scandal should give us pause for reflection on the structures – legal, normative, technological – that created the choice architecture for the postmasters and sub-postmasters.
After being exonerated, a former sub-postmistress said (paraphrased): “Horizon was just software; it wasn’t software that hurt people. It was people who chose to hurt people.”
The pursuit of legal action without unpacking the technology appears to have been a conscious choice by the leaders at the Post Office at the time, within the choice architecture shaped by the same legal and normative structures, but with greater relative ability to pay for lawyers.
We urgently need to question the fitness for purpose of these structures and choice architectures shaped by them.
Legal experts calling this latest development “an exception” will do well to look ahead — to the speedy adoption of black box AI tools in the corporate sector. That will inevitably lead to yet-unforeseeable issues of discrimination and unexplainable decisions. When those disputes come to court, which they inevitably will, the distinction between software and people will be harder to establish.
Recently, on the Nurole Enter The Boardroom podcast, I was asked for my most significant professional insight. I said: .. complex problems of the future will not be solved by applying narrow functional and disciplinary training and tools and approaches of the past. Complex problems require accepting that we will have to do some left field thinking.
Both Horizon – and what is on the horizon – suggest that insight holds good. We need to act accordingly.
(Disclaimer: These are my own views and do not reflect the views of the boards of JP Morgan US Smaller Co.s Investment Trust or Temple Bar Investment Trust or Harmony Energy Income Trust, or Witan, or the Royal Academy of Engineering, where I serve as a non-exec director, and chair or serve on various committees at the time of writing.)