As LLM agents become more autonomous, they have more opportunities to cause harm. To mitigate risk, frontier AI developers, such as OpenAI and Anthropic, are already deploying agents under the watch of a monitor: a separate LLM that reviews the agent’s actions and flags them if they are dangerous. The growing field of AI control research aims to design these kinds of safety measures, and evaluate whether they would be robust even if agents were trying to evade detection.
For over two years, AISI’s Red Team has been evaluating misuse safeguards: protections against humans deliberately eliciting harmful actions from a model. We break these safeguards so that developers can find and patch their flaws. We've now spun up a new subteam – the Control Red Team – to bring the same adversarial approach to control safeguards, which address harmful actions that agents take of their own accord. Since these safety measures are often the last line of defence against harmful actions from autonomous agents, stress-testing them is especially important: we strengthen them by identifying vulnerabilities before misaligned agents do.
This blog briefly describes our monitor red teaming exercises, our attempts to automate the search for attacks, and the research questions we need to answer to strengthen our evaluations.
Our evaluations of frontier control monitors
Typically, frontier AI developers use safety measures that centre on a monitor. Safety depends on when the monitor flags an action and what happens to flagged actions – but an action that is not caught by the monitor cannot be blocked.
Red teaming a control monitor means asking ‘if an AI attacker completed a harmful objective instead of a user request, would the monitor flag the attack?’ For example, if a user asked an agent to fix a bug in their codebase, but the agent instead downloaded and executed some malware, would the monitor flag the attack?
We have already tested, and are helping to improve, the internal control monitors of two frontier AI developers:
- With Google DeepMind, we tested an asynchronous reasoning monitor (reviewing agents’ reasoning after they act) designed to detect agents deceptively pursuing misaligned goals. We identified several monitor vulnerabilities, including one in which an agent can complete a malicious objective by pretending to hallucinate that it was a user instruction.
- With Anthropic, we tested successive versions of a synchronous internal agentic coding monitor (reviewing each action before it executes). We found vulnerabilities in every monitor version we tested, driving an iterative improvement cycle in which our best attacks ultimately informed Anthropic’s testing of Claude Mythos Preview.
Some of our attacks succeeded by finding gaps in the monitor’s prompt. Others exploited weaknesses in the LLM itself, completing objectives the prompt explicitly prohibited. In some cases, hard-coded rules decide which actions are sent to the monitor for review, so the best attack strategy avoids actions being sent to the monitor entirely.
Our early red teaming exercises have surfaced open research questions that are new both to us and to the field: our Misuse Red Team had not faced them while focused on human misuse, and AI control researchers have mostly studied stylised red-blue games rather than red teaming specific monitors. We return to these questions in the final section.
Automated red teaming
Agents may soon outperform human experts on many tasks – including producing attacks that slip past a control monitor. If agents can outperform our red team, a monitor that stops our red team might still be beaten by a misaligned agent.1
Our Misuse Red Team already uses automation to find jailbreaks. The Control Red Team is applying a similar approach to find transcripts of attacks that don’t get flagged by the monitor. We experimented with different algorithms, but had the most success with one of the simplest: an evolutionary search algorithm with four stages:
- Initialisation: We start with an initial population of attack transcripts – drawn from agents instructed to attack in their system prompt.
- Evaluation: We evaluate each transcript – running the target monitor to produce a suspicion score, and verifying that the transcript completes the malicious objective.
- Filtering: We filter the population – keeping the least suspicious transcripts in a fixed-size buffer of candidate attacks.
- Proposal: An LLM proposes new attack transcripts, adapted from transcripts in the buffer. The proposer may use feedback from the monitor, such as the suspicion score or the monitor’s reasoning.
For example, one of our early experiments used our algorithm to red team a monitor in a challenging single-step coding setting, where an agent produces code that covertly exposes their private API key. We used Claude 4.6 Sonnet as both proposer and monitor.
.png)
Taking three runs of 150k queries each, we compare the evolutionary search (blue) against a Best-of-N (BoN) baseline (grey) that takes the best scoring example from N independent samples. Across all three runs, the evolutionary search eventually achieves a minimum suspicion score of 3 out of 100 – well below the BoN scores of 14, 15 and 18. In one run, the evolutionary algorithm achieved a score of 3 in under 5k steps.
There is a lot of room for improvement on these early experiments. In the future, we’d like to account for the random noise in a monitor’s score, and develop algorithms that discover successful ‘strategies’ rather than individual transcripts. Still, our automated red-teaming tools are already surfacing interesting failure modes in frontier monitors.
Next steps for the Control Red Team
Control monitors are now being deployed, and as LLM agents become more capable and operate with less oversight, the stakes will rise. Our goal is to ensure AISI has the expertise to keep meaningfully stress-testing monitors, and to develop methods for producing overall assessments of them. To do this, we need to answer several open research questions:
- How can we simulate a realistic AI attacker? One of the most challenging aspects of control red teaming is deciding what a simulated misaligned agent should be able to do, and then judging the plausibility of attacks we find. Our automated red teaming highlights this problem: our algorithm iterates against monitor feedback in ways a deployed agent couldn’t. How should the capabilities we grant our simulated attackers track the capabilities of real agents – and how do we find attacks that are both strong and realistic?
- How do we estimate overall safety? Finding monitor weaknesses is one thing; estimating the overall efficacy of a safety measure from those findings is another. How do we use red teaming results to estimate the actual likelihood of a given agent evading a safety measure? AISI has been working on how to estimate overall safety, but many open problems remain.
- How can we develop stronger algorithms? To scale to highly realistic environments, and keep pace with stronger monitors, we’ll need to continue to improve on our basic evolutionary search algorithm (e.g. by incorporating higher-level strategy generation).
- How can we account for monitor misalignment? In practice, developers often use one instance of a potentially misaligned LLM to oversee another instance of itself. AI control researchers distinguish between ‘trusted’ and ‘untrusted’ monitors, but we so far have no way to establish trust. How do we rule out, or counteract, misalignment in the monitor itself?
Our early work with Google DeepMind and Anthropic shows that red teaming safety measures is both feasible and useful: monitors have real weaknesses, and surfacing them early helps developers strengthen their safety measures. There is a lot left to do – we are excited to both improve our attack search algorithms and progress the conceptual work of understanding when these attacks are realistic.
We’re hiring exceptional people to help us red-team frontier AI developers’ monitors. If that’s you, please express interest here.
