Shadow-Boxing with AI Safety
how the movement ignores military dangers
Sometimes there may be an impression that the drivers of AI progress care only about its positive effects, and aren’t cautious enough about potential harm. Actually, at least on the face of it, this is not true. There is really a lot of talk about AI safety, it has become one of the big subject areas in AI conferences and most of the high-profile tech leaders, as well as governments, speak all the time about how they’re working to make sure AI is safe. Very few people would openly say that we should continue to develop it without any fear of negative consequences. On one hand, this is encouraging: most people seem to have a, mostly genuine, ethical sense. On the other hand, the safety efforts this work leads to do next to nothing to reassure me about the future. I don’t think anyone (or almost anyone) will deliberately or carelessly build things that are unsafe–the issue is not a lack of effort from the research community. The issue is that the effort has been misappropriated by a mix of causes of secondary importance, and borderline fantasies. As happens in many parts of life, the ‘what’ stage was rushed through, and the field devoted all its energy to the ‘how’ stage. That is, a lot of work has gone into solving problems under the umbrella of AI safety, without properly considering what the big problems are. The result is that the most dangerous applications of AI have been largely ignored by the AI safety movement.
If you seriously ask the question, what’s the most harmful, dangerous, scary thing that AI could be used for, one big answer comes to mind: military usage, building AI-based weapons. Common targets of AI safety research include driverless cars that make a mistake and potentially cause an accident or a medical misdiagnosis–at the very worst these could cost a few lives, whereas AI weapons mean building machines specifically designed to take lives. A failure by a machine that’s supposed to save lives is less sinister, and likely to cause far fewer deaths, than a machine designed to kill being even partially successful. And yet the AI world loves talking about how it’s solving the former, and virtually never mentions the latter.
In other words, the AI safety movement is engaged in a self-imposed narrative control, a kind of shadow boxing. Imagine you have one enemy who is real, scary and powerful, and another, also somewhat real, but less powerful and threatening. Taking on the big enemy is dangerous, and you risk alienating yourself by opposing them and their allies. So instead, you make a show of taking on the less powerful enemy, and you tell yourself that you did something good. That’s not exactly untrue, but the goodness is nested in a broader dishonesty.
Misdirection of AI Safety Work
First, let’s say a bit about these less powerful enemies, the shadow-boxing partners. Of the subfields of AI safety (yes, it’s a big enough field of research that it has well-defined subfields), some are clearly misguided, like the angle that ties interpretability (trying to understand what’s going on inside AI models) with safety. This is based on the unhinged idea that AI systems can learn to deceive or disobey humans. Other AI safety subfields do address a real problem, but one of minor importance—maybe how a chatbot assumes a doctor is a man or some other political bias. A charming example I heard at a conference presentation recently is what if a user asks how to bake a cake, and the recipe from the chatbot contains nuts, but the user is allergic to nuts!! Amongst these fanciful and peripheral problems, there is also mention of preventing misuse by ‘bad actors’--which is a step towards acknowledging the potential for AI weaponisation. But when you read more into it, it fizzles away long before getting to the crux of the matter. Firstly, it focuses almost entirely on chatbots, constructing semi-plausible scenarios in which they become a key military technology, like someone using them to make a bioweapon. Given that other technologies like autonomous vehicle control, remote sensing and object detection have much more direct uses in weapons, the hang-up on chatbots comes off more like a symptom of LLM hype than a serious consideration of the ways AI might be used to deliberately kill people. Secondly, the so-called ‘bad actors’ are not specified in any detail, but it’s clear they’re not thought to include, for example, the US Department of Defence. A vague outline is given of some rogue terrorist organisation, or just a lone mad individual who gets Claude to reveal the steps for making anthrax, while the group in the world that’s spending more money than anyone else on using AI to make weapons is ignored.
There are voices talking about the topic, stopkillerrobots.org, for example, is even named after it, but within the AI world, and amongst big companies and governments, AI safety has come to mean a certain set of things, and military use is not one of them.
Foreignness of Military Culture
Why is there more talk about the dangers of an inaccurate medical system and stopping autonomous cars from crashing than about AI weapons? As a researcher, there’s a definite sense that you shouldn’t bring up the military, and that you’d risk making people uncomfortable if you did. But why? Short of being explicitly told, which I’ve never seen happen, where does the aversion come from?
Partly, it’s the cultural divide between research and the military. Silicon Valley, and by extension most of industrial AI, has an entrenched opinion of technology, which does not sit easily with its use in the military. Fundamentally, Silicon Valley is all about moving onwards and upwards, and sees technology as a tool that can usher in an exciting future. Academia, to a lesser extent, shares this focus on progress, and the belief in an undercurrent of things getting better. The military, on the other hand, seeks to avoid or pre-empt crises, and spends its time planning for various doomsday scenarios around the world. This, and other related differences, mean the two fields see the world in quite distinct ways, and it’s much easier to ignore a subject if it doesn’t resonate with the worldview of those in your circle.
The Strings of Military Funding
Another reason, a less savoury one, is that some AI researchers receive military funding. Depending on how you count, it may actually be that many do. Because it turns out that this cultural divide does not stop the AI world accepting 10s if not 100s of billions of dollars from the military-industrial complex. Some of this comes in the form of Department of Defence (DoD) contracts, those traditionally gobbled up by the likes of Lockheed Martin. Giants such as Amazon and Microsoft receive DoD contracts, and so do a host of military AI startups [1], who attempt to spin Silicon Valley’s general distaste for the military as a dogma which needs to be ‘disrupted’. The money also comes in the form of research grants to universities and other institutes. The DoD spends ~12B annually on basic research1 . What specific things they research is classified, to stop other countries trying to infer their strategy, but given their prioritisation of the area, AI research has to be a substantial chunk of it. And that portion really is just for pure research stuff, there’s separate allocation, another ~80B, on deploying it into military applications. Most stuff in the big AI conferences could be funded as basic research here. China maybe spends ⅓ as much as the US2 . There’s a tight lid on the details about what this goes towards, we could roughly assume both countries spend a similar fraction on research and AI research.
Where does this money go? It’s certainly possible to find some AI research papers with explicit support from e.g. DARPA (an agency under the DoD), and I suspect there are many more that downplay the connection (you can put down whatever you want for funding sources, they’re never checked.) Either way, you’re hardly going to acknowledge military use as a potential harm of your research, if it was specifically funded by the military because they were interested in using it.
This reason for avoiding the topic also has a second-order effect. Even if this particular project doesn’t have military funding, maybe others in the same lab or company do, and they’ll be anxious to protect their funding sources, so they could be worried or annoyed if you talk about the harm of AI weapons. The large tech companies have a formal internal review, where any research published under the company name is vetted by various people to ensure it conforms to the company’s agenda. Possibly (I don’t know), companies like Microsoft and Amazon that receive billions annually from the DoD would remove talk of AI military applications from their papers. In practice, I expect they don’t have to, as the researchers would choose to make their lives easier and just not mention it in the first place. Why risk upsetting your colleagues when you could instead choose an uncontroversial, apolitical alternative like misdiagnosis or malfunctioning driverless cars?
The anthropologist Hugh Gusterton said that when research that could be funded by neutral civilian agencies is instead funded by the military, knowledge is subtly militarized and bent in the way a tree is bent by a prevailing wind (I’d contest that non-military government funding is completely neutral either, but that’s another point). This panopticon-style self-censorship is one way military money warps the research it funds.
Moral Ambiguity
I don’t mean, exactly, that the military is an evil force corrupting AI development. It’s naive to dismiss all military activity as unethical, to say it’s merely the result of power-hungry warmongering. There are some reasons why military research may be appropriate, or even necessary. For the past 80 years, the US military has been an unquestionable global hegemon, and this is part of what’s allowed the West to live in unprecedented peace and security. It’d be disingenuous to now claim the moral high ground from this position, without ever thinking about the reality of war. This US supremacy is not as complete as it was 20 or 30 years ago, and could even be coming to an end in some parts of the world. To take one example, it’s a very real possibility in the next five years that China will invade Taiwan. In the last decade, especially since Xi Jinping came to power, the Chinese Navy, the PLAN, has far outpaced any other country’s, and it has become the largest in the world by a wide margin. Were it not for the US, China would very likely defeat even the combined forces of everyone else in the western Pacific, including Japan, South Korea, Australia and of course Taiwan itself. Among the countless analysed scenarios of how an invasion could play out, the devil is often in the details. The outcome hinges on just how good each side is at many facets of waging war–their intel, their ship speed and stealth, their cyber attack and defence etc. Maybe, if AI can give the US and their allies an edge in these areas, it will shift the predicted outcomes away from anything favourable for China, and deter them from invading in the first place. That is certainly an argument some people make in favour of military research. On the other hand, maybe advancing US military capability is going to make China feel more threatened and push them to double down on their own militarisation. Maybe the rapid expansion of the PLAN is motivated by self-protection, and a determination not to repeat what they call the century of humiliation, beginning when the West used its then vastly superior navy to bully and exploit China for financial gain. My instinct is that building more powerful weapons is a race to the bottom, and the brave decision is to give the other side a chance to do the right thing. But then again, if I lived in Taiwan, if all my friends and family and life were there, then maybe I would feel differently.
Notice the moral ambiguity in this problem, and how much more difficult that makes it to work with. Some people, including many who gravitate to technical fields like AI research, would prefer to stick to engineering problems, where there’s a clear right answer. To some extent, that’s ok, but they have to then admit that they don’t have any say in how their work will affect the world. They’re essentially a pawn in the hands of whoever decides what topic they work on, and that’s normally whoever supplies the money. AI safety, as it currently stands, allows the AI world to feel that they do have control over what they’re creating. Of course, they say AI safety is a hard problem and requires more work, but they feel they have basically pinned down what they need to do to avoid things turning out badly and it is at this point an engineering problem. For those AI researchers that are not comfortable with being a pawn in the game, the right place to begin is with the high-level questions of what we want AI to be, and what we don’t want it to be, and clearly, the ethics of building killing machines is a big part of these questions. This means spending the time to understand parts of the world outside your familiar culture, broaching topics that make people uncomfortable, and tackling morally ambiguous questions that can’t be solved as cleanly as technical ones can. AI safety blocks people from doing these things, because it gives the illusion that the matter is already being dealt with. That’s the whole purpose of this sort of shadow-boxing: to allow people the comforting but false belief that they’re wrestling with the big issues. Oh, you’re concerned about how AI is shaping the world? Great, join the AI safety team, we’ve already identified the key areas for you to work on. We even have metrics and benchmarks and datasets, so just engineer a way to make one of these scores higher and you’re doing your bit to make AI safe.
With hindsight, 2022 was the year when fear began to become a big part of what I feel about AI. This wasn’t because of ChatGPT, and certainly wasn’t because of the claims that it could soon outsmart us and escape our control–it was because of the announcement by the Biden administration that GPU exports to China would be restricted for national security reasons3. This at once revealed the extent of the DoD’s intention to accelerate its militarisation of AI, and stoked the self-fulfilling belief in the AI arms race between the US and China. AI is a subject that I came to out of a quasi-spiritual impulse to understand the nature of the mind and the self, and it’s now getting roped into the most powerful and destructive systems on the planet, including the military-industrial complex, and, potentially, the outbreak of the next major global conflicts. Trying to navigate this central and rapidly changing position brings a host of new questions–questions that are unfamiliar, controversial and ambiguous–and so far the AI world has barely found the courage even to ask them.
This is inferred by taking the known fraction spent on research from 2022, (https://ncses.nsf.gov/pubs/nsf25301, with the fact that total military spending has increased 25% since then https://www.sipri.org/databases/milex.
That is, China’s total military budget is reportedly ⅓ that of the US https://www.sipri.org/databases/milex.
Well done