From the perspective of software engineering, current AI systems are unmanageable, and as a consequence their use in serious contexts is irresponsible. For foundational reasons (rather than any temporary technology deficit), the tools we have to manage complexity and scale are just not applicable.
By ‘software engineering’, I mean developing software to align with the principle that impactful software systems need to be trustworthy, which implies their development needs to be managed, transparent and accountable. I don’t suggest any particular methodologies or tools (before you know it, I’d have someone explaining why ‘waterfall’ is wrong!) — but there are some principles which I believe to be both universal and unsatisfiable with current AI systems.
When I last gave talks about AI ethics, around 2018, my sense was that AI development was taking place alongside the abandonment of responsibility in two dimensions. Firstly, and following on from what was already happening in ‘big data’, the world stopped caring about where AI got its data — fitting in nicely with ‘surveillance capitalism. And secondly, contrary to what professional organisations like BCS and ACM had been preaching for years, the outcomes of AI algorithms were no longer viewed as the responsibility of their designers — or anybody, really.
‘Explainable AI’ and some ideas about mitigating bias in AI were developed in response to this, and for a while this looked promising — but unfortunately, the data responsibility issue has not gone away, and the major developments in AI since then have made responsible engineering only more difficult.
How neural networks work
When I say ‘current AI systems’, I mean systems based on large neural networks, including most generative AI, large language models (LLMs) like ChatGPT, most of what DeepMind and OpenAI are producing and developing, and so on. An extremely optimistic view of these is what I would call ‘LLM-functionalism’: the idea that a natural language description of the required functionality fed to an LLM, possibly with some prompt engineering, establishes a meaningful implementation of the functionality.
The neural networks underlying these systems have millions of ‘nodes’ or ‘neurons’. Each has one output and multiple inputs, which either originate externally to the entire network or are taken from other nodes’ outputs. A node’s output is determined by the inputs, the weights put on each input, and an ‘activation function’ that decides how the weighted inputs translate into an output. The connection structure is fixed, using connected ‘layers’ or other structures such as recurrent networks or transformers, as is the activation function. The fixed structure is relevant to what type of problems the network can deal with, but the network’s functionality is almost entirely introduced by ‘training’, which means setting and modifying the weights of each input until the outputs achieve an objective satisfactorily on a set of training data.
The training of the huge networks that make up current AI systems has typically taken an astronomical amount of compute, measurable in the millions of dollars or kWh, and will necessarily have been mostly unsupervised or self-supervised. Put bluntly, it will have required no human input – though there may have been some human tuning afterwards (such as reinforcement learning from human feedback (RLHF) or ‘guard rails’), or when the system runs (such as context and prompt engineering).
Emergence and compositionality
Many of these neural network systems are stochastic, meaning that providing the same input will not always lead to the same output. The behaviour of such AI systems is ‘emergent’ — which means despite the fact that the behaviour of each neuron is given by a precise mathematical formula, neither this behaviour nor the way the nodes are connected are of much help in explaining the network’s overall behaviour.
My first 20 years of research were in formal methods, where mathematics and logic are used to ensure systems operate according to precise formal specifications, or at least to support verification of implemented systems. Software engineering, and particularly formal methods, has not been as successful in managing emergent behaviour or even the aspects of ‘traditional’ systems that have emergent tendencies such as resource usage or security. This is for foundational reasons rather than for a lack of scientific effort.
