Using AI responsibly

Black boxes and lack of transparency continue to be a major problem. If we won’t be able to figure out why a harmful effect happened, we consequently can’t fix that issue.

– AI Researcher Wendell Wallach, Yale University

There is a particular challenge with more complex AI systems called the “black box”. When the datasets that are used are massive, possibly composed of several sources and contain thousands of data points, the reasoning made by the algorithm from the dataset is not transparent and comprehensible. We can only conclude that “the machine produced what it did”.

This is a major problem in terms of reliability. Especially in the public sector, for the sake of good governance, an AI system should work in a way that we can justify and explain its results. This is important just from the perspective of citizens’ right of appeal and to claim damages.

Read about the Algorithmic Transparency Recording Standard on the GOV.UK websiteOpens in a new window..

Some have disregarded the black box problem by for example saying that we cannot fully explain how the brain of a human or a dog functions, but that does not stop us. But biological, autonomous creatures are not owned by anyone, and no-one programs objectives for them, unlike data processing systems. Human consciousness and our right to autonomy hamper this analogy even further.

It has also been suggested that it is acceptable for a black box AI to sometimes produce incorrect results if they cause minor harm compared to the benefits of the correct results. One criticism of this argument relates to medical ethics: would it be acceptable to use a method that is likely to cause harm to 5% of patients?

Opening the black box is an important goal that data science is constantly working on. And we have to keep at it. However, as long as the black box remains a problem, organisations using AI systems must seriously consider what the analyses, decisions or forecasts produced by this kind of AI can be applied to.