The Pentagon’s AI war machine

Project Maven, initiated by the Pentagon in 2017, aimed to integrate artificial intelligence, specifically computer vision, into battlefield operations. The primary goal was to enhance the analysis of drone feed video footage, particularly in the context of ongoing counterterrorism wars. However, it also represented a broader strategic effort to modernize the U.S. military's approach to warfare, recognizing a perceived lag behind commercial technological advancements, especially in areas like automated systems and AI. The project was championed by then-Deputy Secretary of Defense Bob Work and led on the ground by Marine Colonel Drew Cukor. Cukor, drawing from his experiences in Afghanistan where insufficient information had endangered Marines, envisioned AI as a tool to provide front-line operators with critical intelligence, aiming to make warfare safer and more efficient. He acknowledged that AI could displace human roles and challenge existing military doctrines, but believed it was essential for supporting those in harm's way. Early deployments of Maven, particularly in Somalia in late 2017 and early 2018, faced significant challenges. The AI algorithms were reportedly unreliable, leading to user frustration and abandonment of the system. The small team behind Maven worked to overcome these issues, focusing on improving the algorithms and encouraging adoption through personal relationships with commanders and operators, especially within Special Operations Command. A key breakthrough occurred around 2018-2019 when AI reportedly helped identify Marines caught in an ambush in Afghanistan by perceiving through smoke, a scenario where distinguishing friend from foe was critical and difficult for human eyes. This success began to build confidence in AI's potential. A significant point of discussion revolves around the concept of "human in the loop." While military leaders often stated that humans would always retain control, the Pentagon's official directives on autonomy, updated in 2023, speak of "appropriate levels of human judgment over the use of force," a phrasing that allows for interpretation and supervision rather than direct, moment-to-moment human control. This ambiguity has become a focal point in discussions about AI's role in lethal decision-making. The initial memo for Project Maven focused on drone footage for counter-ISIS operations, but the intention behind its development, particularly regarding targeting, has been a subject of debate. Colonel Cukor maintained that targeting was always a consideration from the outset, envisioning a system where precise coordinates could be identified and weapons directed with greater speed and accuracy. This vision aimed to streamline and potentially bypass existing, complex intelligence and targeting processes. The potential for AI to make warfare more efficient has also raised concerns. While Cukor framed his motivations around saving civilian lives and protecting friendly forces, other individuals within the project anonymously expressed a desire to "kill people all the time" using AI. This highlights a divergence in perspectives, with some seeing AI as a tool to accelerate and scale lethal operations against perceived enemies. The integration of AI into military systems has accelerated significantly. Centcom is employing various AI tools, including Maven Smart System, which can process a vast amount of data. With the help of Large Language Models (LLMs) like Anthropic's Claude, the system can now identify thousands of potential targets daily, significantly speeding up the administrative processes involved in targeting packages, though human and legal review remain crucial. The development of autonomous weapons is a key area of focus. While most current drone operations still involve human pilots, initiatives like "Replicator" under the Biden administration aim to deploy large numbers of affordable, attritable, and potentially autonomous drones. The ambition is to create swarming drone systems capable of detecting and engaging targets autonomously, drawing on algorithms trained on specific data, such as identifying Chinese vessels in the Indo-Pacific. Challenges remain in integrating these systems and ensuring operator trust. The debate over AI's role in warfare is increasingly public, as seen in the conflict between Anthropic and the Pentagon. Anthropic, despite being an early adopter of classified cloud systems for national security, has expressed red lines against mass domestic surveillance and fully autonomous weapons without human oversight. This stance has led to friction and has highlighted the Pentagon's determination to acquire the AI tools it deems necessary, even if it means exploring alternative providers like XAI and OpenAI. The concept of "acceptable risk" is central to AI deployment. Experts acknowledge that AI is a "black box" technology with inherent risks like hallucinations, bias, and algorithmic drift. The level of acceptable risk appears to be context-dependent; for instance, AI use in a maritime combat scenario against a state actor might be deemed less risky in terms of civilian casualties than in an urban environment. However, the potential for errors, such as misidentifying targets or striking friendly forces, remains a concern. The bombing of a school in Iran, where AI systems were reportedly involved in identifying targets, has brought the risks of AI in targeting to the forefront. While the US is investigating, the incident underscores the critical need for accurate targeting lists, up-to-date intelligence, and robust cross-referencing with open-source information, such as Google Maps, to prevent tragic errors. The failure to do so, even with AI capabilities, can lead to devastating consequences. The moral implications of AI in warfare are profound. Some express deep discomfort with anything that makes killing easier, faster, and cheaper, arguing that war should remain a difficult and costly human endeavor. There is also a fundamental uncertainty about humanity's ability to control advanced AI systems, raising further alarm. The increasing comfort levels of technologists in engaging with military applications of AI are seen by some as a testament to the Pentagon's influence, shifting the discourse around what is considered permissible in the pursuit of national security. The migration of these powerful tools from the battlefield to domestic law enforcement and surveillance also remains a significant societal concern.