What would we do without science-fiction futures? It would certainly be a lot harder to explain the potential threat of artificial intelligence to human society without the armoury of doomsday storylines that have permeated the genre since Karel Čapek gave us the word 'robot' for obedient smart automaton, which went on to kill off most of humanity.
If we are not hunted by the self-aware Skynet in the Terminator franchise or turned into the most inefficient batteries ever conceived we are imprisoned forever for our own protection only to be saved because it turns out that despite being superintelligent, the machine blows a fuse when it is introduced by some pesky human to some logical incompatibility that sends it into an infinite loop.
If you were writing open letters and giving interviews on the risks of AI, you would find it tough to avoid painting these pictures. Most of the people writing and signing open letters calling for controls over AI are focused on the sci-fi future rather than the ugly reality of what the current generation of systems are capable of. In his own personal statement, Yoshua Bengio, one of the deep-learning pioneers, argued the near-term risks are easier to predict, which is why the long-term issues are more important.
The open letter published in late March by the Future of Life Institute, which called for a moratorium beyond GPT-4 for "at least six months", had an objective that is easily stated but which is profoundly hard to realise: "Powerful AI systems should be developed only once we are confident that their effects will be positive and their risks will be manageable". It is an entirely understandable and a worthwhile objective but it is also a comparatively easy sell that if fixed carelessly or superficially leaves a lot of room for abuse – and potentially just as big a threat to society.
Who isn't going to think that's worth signing up to (who isn't a sociopath)? It also easily becomes hostage to claims of national security and how the benefits of medical AI outweigh the negatives of systems that subvert society.
But is the problem that AI is getting too advanced or that it is nowhere near advanced enough? Has it just been given enough computer power to convey the illusion of competence? The FoLI letter contains the frequently made claim that "AI systems are now becoming human-competitive at general tasks". This is true, if you squint hard enough.
In practice, these systems have been competitive at certain tasks only if the tasks have been presented in a way that flatters the machine, whether deliberately or accidentally. It has been a decade since image-recognition systems outpaced humans on reading road signs, even those that had been bleached to a near uniform white by the sun. It turned out digital cameras record information the human eye and brain will ignore that provided important clues to the fact that the sign used to display a speed limit. Similarly, GPT-class systems can assemble a huge amount of data to produce viable summaries. But with the wrong prompt the machine will confidently assert the existence of sources and references that were never written.
There is quite a mismatch between the rhetoric around "powerful AI" and the work going on inside the research institutes and corporations. One major issue is that there is something of a symbiotic relationship between those who want to understand the mammalian brain and those who use the brain as inspiration for providing computers with the ability to learn. There is nothing particularly wrong with this approach given the difficulty of understanding intelligence simply by observing the behaviour of biological brains and, conversely, the many blind alleys AI research has pursued. Even though it is quite far removed from the way in which brains seem to work, the deep neural network (DNN) has undoubtedly proved immensely successful.
But numerous researchers point to the huge gap between the comparatively simple architecture of the DNN and the sprawling complexity of the brain. Many relate the ability of the DNN to what psychologist Daniel Kahneman calls System 1, the instinctual, unconscious reaction to inputs, and that even System 1 might be beyond today's systems. It might be little more than the basic Perceptual response that has us flinch away instantly from an object flying towards us. System 1 can at least understand simple sentences but it's still unclear as to whether any of these systems comprehend meaning rather than identifying patterns in how words are used.
The idea that AI should reason much about anything has largely been left on the back burner while researchers try to work out how it might work in practice or just hope that continuing tweaks to DNN might magically result in basic reasoning. But in this environment, the machine cannot logic itself into oblivion when faced with the ultimate conundrum because it simply had no notion of reason in the first place. It simply had the notion and responsibility projected onto it by people. Instead, it stitches together answers that much of the time look vaguely right but which as often are imperceptibly wrong. And the results could be disastrous if we hand the reins to a machine that seems to be up to the task but suddenly makes incomprehensible decisions because of gaps in the training. The car crashes simply because the machine has hallucinated a situation that can never exist.
Those near-term risks are real and to some extent happening. The enormous popularity of ChatGPT, with its billion users per month according to recent statistics, show how willing people are to cede control to machines that are not quite up to the job. We just do not see the problems, because only a few companies have implemented AI in control systems rather than seemingly less dangerous chatbots.
A decade ago, computer-science pioneer Tony Hoare proposed an updated Turing Test: rather than engaging it in conversation, a machine should only be considered intelligent if it could explain its own program. His proposal was more focused on programming but it could provide a better foundation for assessing AI than today's cobbled-together benchmarks and an arguably higher hurdle to pass than the informal assessment that seems to lie at the heart of today's call for a moratorium: "Powerful AI systems should be developed only once we have a handle on what intelligence actually involves and so can gauge whether any implementation does not convey excessive risks".
