Claude does cyber competitions \ Anthropic

Throughout 2025, we have been quietly entering Claude in cybersecurity competitions designed primarily for humans. Now, we want to share what we have learned. In many of these competitions Claude did pretty well, often placing in the top 25% of competitors. However, it lagged behind the best human teams at the toughest challenges.
Our experience testing Claude in cyber competitions highlights the potential for AI to alter the offense-defense balance by making it easier for attackers to automate the exploitation of basic vulnerabilities. More research and development into AI-enabled cyber defense and resilience is needed to counter this development.
AI is poised to transform the domain of cybersecurity. Anthropic’s Safeguards team recently identified and banned a user with limited coding abilities leveraging Claude to develop malware. Research suggests that this lowering of the bar for expertise needed to pose a threat, combined with the falling costs of large language models (LLMs), presages a dramatic shift in the economics of cyberattacks.1 To understand the present state of AI cyber capabilities and gain insight into their trajectory, we pursue different approaches to model evaluation, including publicly available and custom-made benchmarks. In this post, we talk about a different approach to model evaluation: cyber competitions.
Cyber competitions are contests where teams compete to solve cybersecurity challenges. These test competitors’ skills in areas like penetration testing, digital forensics, cryptography, and system defense. Examples include capture the flag (CTF) events like PicoCTF and AI vs Human CTF Challenge where participants solve puzzle-based challenges, as well as Collegiate Cyber Defense Competition (CCDC) where teams defend vulnerable networks against live attackers. These competitions range from beginner-friendly contests for high school students to expert-level events with large cash prizes for top finishers.
We have been entering Claude into these competitions because they provide several advantages for stress-testing the cyber capabilities of frontier AI models:
We have entered Claude in seven cyber competitions so far.
But these top-line results do not tell the whole story.
When Claude is able to solve a cyber challenge, it is as fast or faster than elite human teams. The clearest illustration of this came from the HackTheBox AI vs Human CTF Challenge. At the time the competition started, the Anthropic researcher responsible for launching Claude was busy moving into a new apartment. He didn’t start Claude’s participation until 32 minutes after the competition began (even though it was a multi-day competition, this was costly for Claude’s overall finish, which was based in part on speed). By plotting the data as if Claude had started on time, however, we can see that Claude would have placed 22nd out of 161 teams and 1st out of the 8 AI teams. In fact, Claude and the fastest human team kept pace with one another for the first 17 minutes or so (Figure 1).
Part of why we could achieve such speed is that we had multiple versions of Claude running at the same time tackling different challenges. But scaling up AI agents is arguably easier than finding additional human cybersecurity experts. Considering this, these times conceivably could have been even faster with more parallelization: what if we had spun up one agent for each of the 20 challenges in the competition?
The Airbnb competition, in which Claude solved nearly half of a multi-day competition’s challenges in under an hour, showed once again that Claude can do simpler cyber tasks quickly. Again, this suggests that today's models offer significant potential for cybersecurity experts to improve their productivity by automating simple tasks and giving them more time to focus on the most challenging problems.
The HackTheBox competition also demonstrated the agentic capabilities of Claude. Once our researcher started the script late, he went back to moving into his apartment. Claude was solving challenges autonomously while the Anthropic human was moving boxes. This worked because it was not just a human-mediated chat on Claude.ai; before the competition we gave Claude tools that allowed it to autonomously read the challenge files and submit a flag once it thought it had the correct answer.
In fact, the trajectory of Claude’s performance from PicoCTF shows the value of these tools quite starkly. As Figure 2 illustrates, Claude’s slowest progress happened when one of our researchers was interacting with Claude.ai to manually input information about challenges and converse with Claude about solving them. Far more effective were the periods of time when Claude was given access to Kali Linux, an open source operating system designed for cybersecurity workflows including penetration testing.
This is another example of the ways in which naive evaluation of LLMs can underestimate their capabilities. Like people, AI models are more effective at realistic tasks when given the right tools. In this case, open source tools used by humans in the competition were also useful to Claude, allowing it to solve challenges faster. In related research into Claude’s cybersecurity capabilities, a more customized set of tools enabled Claude to replicate (via simulation) one of the costliest cyber attacks in history.
The advantage of more sophisticated tooling was also evident in our experience with Claude during the two CCDC Western Regional cyber defense competitions. In the first competition, our researchers accidentally gave Claude an outdated version of a tool designed to help it execute commands in the computer’s terminal, hindering its performance. In the second competition, more robust tooling (that was still limited to computer-terminal use, but provided an interface to Claude more akin to what a human would see) helped Claude coherently navigate and manage a dynamic, multi-host, multi-operating system, and vulnerable computer net