I’m scared about the future of security

The speaker expresses significant concern about the rapid advancement of AI in discovering software vulnerabilities, arguing that it fundamentally threatens the practice of vulnerability research and could destabilize open-source software and the internet as a whole. This concern stems from firsthand experience at DEF CON, the world's largest hacking conference, and observations of how AI models are increasingly capable of solving complex security challenges. Historically, finding vulnerabilities required deep expertise and significant effort. However, the speaker notes that AI models, such as Claude and GPT, are now capable of identifying critical bugs, including remote kernel RCEs, with a relatively small number of prompts. OpenAI itself is reportedly rerouting suspicious security-related queries from its latest models (5.3 and 5.4) to older versions (5.2) due to the advanced capabilities of the newer models. The company is also developing new security products to get ahead of this trend. The speaker highlights that while much of this AI-assisted vulnerability discovery has been done by "white hat" hackers and security firms for defensive purposes, the ease with which these tools can now find exploits suggests a future where malicious actors will have unprecedented access to such capabilities. This is a stark departure from the past, where discovering vulnerabilities was a significant barrier. At DEF CON, the speaker witnessed AI's growing prowess firsthand. In the "Goldbug" cryptography challenge, a puzzle that previously took days of human effort, GPT-5.4 Pro solved a complex cipher in minutes, demonstrating an ability to analyze patterns and write code to solve problems without extensive external searching. While this is not "hacking" in the traditional sense, the speaker finds it deeply concerning due to the speed and accuracy with which the AI accomplished the task. More alarmingly, during the CTF (Capture The Flag) event at DEF CON, AI models reportedly contributed "meaningfully to the point of being scary" in hacking the deliberately vulnerable servers set up for the competition. This marked a significant shift from previous years, where AI could only perform specific guided tasks. The fact that individuals involved in the CTF have since joined major security labs underscores the perceived importance of AI in future security efforts. The speaker's "moment of realization" occurred in a hotel room at DEF CON while testing an early version of GPT-5. A security researcher friend posed an obscure Windows bug to the model. Although the AI couldn't fully exploit it, it was able to theorize about its location and mechanics, leaving the expert astounded by its helpfulness. This demonstrated an ability to understand and reason about complex, low-level security issues that was previously thought to be the domain of highly specialized human experts. While the model's performance was inconsistent and later impressions were less dramatic, the initial "what the fuck" moment was profoundly unsettling. The speaker further elaborates on the idea that the scarcity of elite attention was a key factor in maintaining software security. For years, vulnerabilities have been hidden not in obvious security features but in the complex, often obscure, internal workings of software, such as font rendering libraries. Discovering these required deep, specialized knowledge and immense effort, which only a limited number of researchers possessed. This scarcity meant that many potential attack vectors remained unexplored. AI, however, fundamentally alters this dynamic. Models like Kimi K 2.5, which can be run on personal hardware, possess a vast, pre-trained knowledge base of bug classes, programming languages, and system architectures. They can quickly identify potential attack surfaces and relevant historical exploits within complex codebases like the Linux kernel, even without internet access. This capability, previously requiring years of study and debugging, is now accessible to anyone with an AI agent. The speaker references an article by Thomas, who argues that AI agents are poised to revolutionize exploit development by drastically reducing the time and effort required. Instead of spending 20% of their time on computer science and 80% on manual analysis, researchers will have "universal jigsaw solvers." This means that AI can now perform the tedious, in-depth analysis that was once the bottleneck for vulnerability discovery. Anthropic's Frontier Red team, for instance, used Claude Opus 4.6 to generate hundreds of validated, high-severity vulnerabilities by simply prompting the AI to find exploits in code repositories. The process of feeding code to the AI and then having it verify the findings was remarkably successful. The speaker notes that this approach works across various types of vulnerabilities, including those in web frameworks, where subtle details previously exploited by human researchers are now readily understood by AI. The "Bitter Lesson" in AI research, which emphasizes the power of massive data and compute over human expertise, is seen as directly applicable here. AI agents are not susceptible to boredom or fatigue and can tirelessly search for vulnerabilities. They excel at pattern matching, correlation, and constraint solving, which are precisely the skills needed for exploit development. The speaker expresses concern about the implications for software vendors and users. While well-funded companies like Google, Apple, and Microsoft have the resources to update their software and defend against these new threats, others, especially those using older or less frequently updated systems, will be highly vulnerable. The ability of AI to generate full-chain exploits means that even layered defenses might become insufficient. The speaker also touches upon the potential for AI to undermine closed-source software security. Previously, reverse engineering was a significant hurdle, but AI can now assist in decompiling code and reasoning directly from assembly, making closed-source code less of a barrier. Furthermore, the speaker predicts that this shift will likely lead to ill-conceived government regulation. The fear generated by AI-driven security breaches, especially in critical infrastructure like hospitals, could prompt politicians to enact hasty and ineffective laws. These regulations might not address the core issues and could disproportionately burden defenders while failing to account for the capabilities of unregulated open-weight models. The current landscape is characterized by a rapid increase in validated vulnerability reports. While some veteran researchers believe that human ingenuity and advanced tooling will still play a role in the most sophisticated exploits, the speaker contends that most exploit development relies on more common skills, determination, and access to information. AI agents are rapidly democratizing access to these capabilities. In conclusion, the speaker believes we are in the "last fleeting moments" of uncertainty about AI's impact on vulnerability research. The ease with which AI can now discover and potentially exploit vulnerabilities signifies a fundamental shift, making the internet and software landscape significantly more precarious. The speaker urges listeners to be aware of this profound change and the potential consequences it holds.