Killer Robots and Mass Surveillance

This week we talk about Anthropic, the Department of Defense, and OpenAI.

We also discuss red lines, contracts, and lethal autonomous systems.

Recommended Book: Empire of AI by Karen Hao

Transcript

Lethal autonomous weapons, often called lethal autonomous systems, autonomous weapons systems, or just ‘killer robots,’ are military hardware that can operate independent of human control, searching for and engaging with targets based on their programming and thus not needing a human being to point it at things or pull the trigger.

The specific nature and capabilities of these devices vary substantially from context to content, and even between scholars writing on the subject, but in general these are systems—be they aerial drones, heavy gun emplacements, some kind of mobile rocket launcher, or a human- or dog-shaped robot—that are capable of carrying out tasks and achieving goals without needing constant attention from a human operator.

That’s a stark contrast with drones that require either a human controlled or what’s called a human-in-the-loop in order to make decisions. Some drones and other robots and weapons require full hands-on control, with a human steering them, pointing their weapons, and pulling the trigger, while others are semi-autonomous in that they can be told to patrol a given area and look for specific things, but then they reach out to a human-in-the-loop to make final decisions about whatever they want to do, including and especially weapon-related things; a human has to be the one to drop the bomb or fire the gun in most cases, today.

Fully autonomous weapon systems, without a human in the loop, are far less common at this point, in part because it’s difficult to create a system so capable that it doesn’t require human intervention at times, but also because it’s truly dangerous to create such a device.

Modern artificial intelligence systems are incredibly powerful, but they still make mistakes, and just as an LLM-based chatbot might muddle its words or add extra fingers to a made-up person in an image it generates, or a step further, might fabricate research referenced in a paper it produces, an AI-controlled weapon system might see targets where there are no targets, or might flag a friendly, someone on its side, or a peaceful, noncombatant human, as a target. And if there’s no human-in-the-loop to check the AI’s understanding and correct it, that could mean a lot of non-targets being treated like targets, their lives ended by killer robots that gun them down or launch a missile at their home.

On a larger scale, AI systems controlling arrays of weapons, or even entire militaries, becoming strategic commanders, could wipe out all human life by sparking a nuclear war.

A recent study conducted at King’s College London found that in simulated crises, across 21 scenarios, AI systems which thought they had control of nation-state-scale militaries opted for nuclear signaling, escalation, and tactical nuclear weapon use 95% of the time, never once across all simulations choosing to use one of the eight de-escalatory options that were made available to them.

All of which suggests to the researchers behind this study that the norm, approaching the level of taboo, associated with nuclear weapons use globally since WWII, among humans at least, may not have carried over to these AI systems, and full-blown nuclear conflict may thus become more likely under AI-driven military conditions.

What I’d like to talk about today is a recent confrontation between one AI company—Anthropic—and its client, the US Department of Defense, and the seeming implications of both this conflict, and what happened as a result.

—

In late-2024, the US Department of Defense—which by the way is still the official title, despite the President calling it the Department of War, since only Congress can change its name—the US DoD partnered with Anthropic to get a version of its Claude LLM-based AI model that could be used by the Pentagon.

Anthropic worked with Palantir, which is a data-aggregation and surveillance company, basically, run by Peter Thiel and very favored by this administration, and Amazon Web Services, to make that Claude-for-the-US-military relationship happen, those interconnections allowing this version of the model to be used for classified missions.

Anthropic received a $200 million contract with the Department of Defense in mid-2025, as did a slew of other US-based AI companies, including Google, xAI, and OpenAI. But while the Pentagon has been funding a bunch of US-based AI companies for this utility, only Claude was reportedly used during the early 2026 raid on Venezuela, during which now-former Venezuelan President Maduro was taken by US forces.

Word on the street is that Claude is the only model that the Pentagon has found truly useful for these sorts of operations, though publicly they’re saying that investments in all of these models have borne fruit, at least to some degree.

So Anthropic’s Claude model is being used for classified, military and intelligence purposes by the US government. Anthropic has been happy about this, by all accounts, because that’s a fair bit of money, but also being used for these purposes by a government is a pretty big deal—if it’s good enough for the US military, after all, many CEOs will see that as a strong indication that Claude is definitely good enough for their intended business purposes.

On February 24 of 2026, though, the US Defense Secretary, Pete Hegseth, threatened to remove Anthropic from the DoD’s stable of AI systems that they use unless the company allowed the DoD to use Claude for any and all legal purposes—unrestricted use of the model, basically.

This threat came with a timeline—accede to these demands by February 27 or be cut from the DoD’s supply chain—and the day before that deadline, the 26th, Anthropic’s CEO released a statement indicating that the company would not get rid of its red lines that delineated what Claude could and could not be used for, and on the 27th, US President Trump ordered that all US agencies stop using Anthropic tools, and said that he would declare the company a supply chain risk, which would make it illegal for any company doing business with the US government at any level and in any fashion to use Anthropic products or services—a label that’s rarely used, and which was previously used by the Trump administration against Chinese tech giant Huawei on the basis that the company might insert spy equipment in communications hardware installed across the US if they were allowed to continue operating in the country.

Those red lines that Anthropic’s CEO said he wouldn’t get rid of, not even for a client as big and important as the US government, and not even in the face of threats by Hegseth, including that he might invoke the Defense Production Act, which would allow him to force the company to allow the Pentagon to use Claude however they like, or Trumps threat that the company be blacklisted from not just the government, but from working with a significant chunk of Fortune 500 companies, those red lines include not allowing Claude to be used for controlling autonomous weapon systems, killer robots, basically, and not allowing Claude to be used for surveilling US citizens.

The Pentagon signed a contract with Anthropic in which they agreed to these terms, but Hegseth’s new demand was that Anthropic sign a new version of the contract in which they allow the US government to use Claude and their other offerings for ‘all legal purposes,’ which apparently includes, at least in some cases and contexts, killer robots and mass surveillance.

So the Pentagon tried to strong-arm a US-based AI company into allowing them to use their product for purposes the company doesn’t consider to be moral, and that led to this situation in which Anthropic is now being phased out from US government use—it’ll apparently take about 6 months to do this, and some analysts speculate that timeline is meant to serve as a period in which further negotiation can occur—but either way, it’s being phased out and it may even have trouble getting major clients in the future as a result of being blackballed.

As all this was happening, OpenAI stepped in and offered its products and services to fill the void left by Anthropic in the US government.

OpenAI’s CEO has been cozying up to Trump a lot since he regained office, and has positioned the company as a major US asset, too big to fail because then China will win the AI race, basically, so this makes sense. Its CEO released several statements and press releases in the wake of this further cozying, saying that they believe the same things Anthropic does, and that they’re not giving up any credibility for doing this because they have the same red lines, no killer robots, no mass surveillance of US citizens.

But this is generally assumed to be bunk, because why would the Pentagon agree to the same terms all over again, and with a company that provides, for their purposes and right now, anyway, inferior services instead of the one they just chased out and blackballed, and which was helping them do purposeful, effective things, like kidnapping a foreign leader from a secure facility, today?

Instead, what it sounds like is OpenAI is trying to have its cake and eat it too, saying publicly that they don’t want their offerings used to control autonomous weapons systems or mass surveil Americans, but instead of writing that into the contract, they’ve got some basic guardrails baked into their systems, and they are assuming those guardrails will keep any funny business from happening. So it’s a sort of gentleman’s agreement with their clients that OpenAI products won’t be used for mass surveillance or killer robots, rather than something legally binding, as was the case with Anthropic.

The response to all this within the tech world has been illustrative of what we might expect in the coming years. Many people, including folks working on these technologies, are halting their use of OpenAI tech in protest, and in some (at this point at least) fewer cases, people are quitting their OpenAI jobs, because they are strongly opposed to these use-cases and would prefer to support a company that takes a strong stand on these sorts of moral issues.

Some analysts also wonder if this will ensure the Pentagon only ever has access to inferior AI models because they intentionally threatened and disempowered a key AI industry CEO in public, saying that they had final say over how these tools are used, and many such CEOs are both unaccustomed to such stripping down, but are also doing the work they’re doing for ideological reasons—they have beliefs about what the future, as enabled by AI technologies, will look like, and they believe they will play a vital role in making that future happen.

The idea, then, is why would they want to work with the Pentagon, or the US government more broadly, if that means no longer being in charge of the destiny of these tools they’re putting so much time, effort, and resources into building? Why would they take on a client, even a big, important one, if that means no longer having any grain of control over the future of the world as shaped by the systems they’re building?

We’ll know a bit more about how all this plays out within the next handful of months, as this could serve as a moral differentiator between otherwise near-match products in the AI category, allowing companies like Anthropic to compete, both in terms of clients and in terms of employees, with the likes of OpenAI and xAI by saying, look, we don’t want killer robots or mass surveillance and we gave up a LOT, put our money where our mouths are, in support of that moral stance.

That could prove to be a serious feather in their cap, despite the initial cost, though it could also be that the pressure the US government is willing and able to apply to them instead serves as a warning to others, and the likes of OpenAI and Google and so on just get better at speaking out of both sides of their mouths on this issue, creating sneakier contracts that allow them to say the same on paper, seeming to take the same moral stance Anthropic did, while behind closed doors allowing their clients to do basically whatever they want with their products, including using them to control killer robots and to mass surveil US citizens.

Show Notes

https://www.kcl.ac.uk/news/artificial-intelligence-under-nuclear-pressure-first-large-scale-kings-study-reveals-how-ai-models-reason-and-escalate-under-crisis

https://www.axios.com/2026/02/26/ai-nuclear-weapons-war-pentagon-scenarios

https://www.nytimes.com/2026/02/27/technology/openai-agreement-pentagon-ai.html

https://en.wikipedia.org/wiki/Lethal_autonomous_weapon

https://www.theverge.com/ai-artificial-intelligence/885963/anthropic-dod-pentagon-tech-workers-ai-labs-react

https://www.theverge.com/ai-artificial-intelligence/886816/openai-reached-a-new-agreement-with-the-pentagon

https://arstechnica.com/tech-policy/2026/02/trump-moves-to-ban-anthropic-from-the-us-government/

https://apnews.com/article/anthropic-pentagon-ai-dario-amodei-hegseth-0c464a054359b9fdc80cf18b0d4f690c

https://www.wsj.com/tech/ai/whats-really-at-stake-in-the-fight-between-anthropic-and-the-pentagon-d450c1a1

https://openai.com/index/our-agreement-with-the-department-of-war/

https://www.kcl.ac.uk/news/artificial-intelligence-under-nuclear-pressure-first-large-scale-kings-study-reveals-how-ai-models-reason-and-escalate-under-crisis

https://www.axios.com/2026/02/26/ai-nuclear-weapons-war-pentagon-scenarios

This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit letsknowthings.substack.com/subscribe