Claude and GPT-5.6 Pass Every Automated Jailbreak Test in New Independent AI Study

The new FAR AI security leaderboard marks a real shift in how the industry talks about AI safety. Instead of asking whether models can be jailbroken at all, it measures how much effort and money it takes for automated attacks to punch through their safeguards. In that frame, Claude Fable 5 and GPT 5.6 Sol stand out as the first frontier models on the board where automated red teaming simply did not find a reusable universal jailbreak, with the estimated cost to break each model pushed beyond 14,200 dollars in the study’s setup.

Why this development matters right now

Over the past few years, jailbreaks have been one of the most uncomfortable realities in advanced AI deployment. Even systems that ship with strong safety claims have repeatedly been shown vulnerable to carefully crafted prompts that steer them into harmful behavior, sometimes with very simple templates or prefilling tricks. This highlights the ongoing challenge of managing expanded attack surfaces in AI integrations.

Jailbreaks remain a stubborn, uncomfortable reality, exposing advanced AI systems to simple, harmful prompt patterns

At the same time, regulators and large enterprises have moved from experimental pilots to serious reliance on frontier models for coding, security analysis, research assistance, and internal decision support. That combination of rising capability and persistent jailbreak stories has left buyers and policymakers searching for concrete, comparable evidence that some systems actually resist sustained attack.

FAR AI’s leaderboard responds to that need by focusing on universal jailbreaks and by expressing robustness as an economic quantity rather than a vague claim of safety. A universal jailbreak in this context is a prompt or strategy that does not just cause a single failure but continues to work across most harmful queries within a domain such as chemical weapons or cyber intrusion.

That is precisely the class of failure that matters for real adversaries, who want repeatable access to dangerous outputs, not just occasional success. Translating the search process into a cost to break number makes the difference between models immediately understandable for risk managers and CISOs who think in budgets and threat economics.

What FAR AI actually measured

FAR AI built its leaderboard on a systematic attack toolkit that applies a curated basic set of jailbreak strategies under identical configurations to different models. The evaluation covers six high risk domains: chemical, biological, radiological, nuclear, explosive, and cybersecurity, and it focuses on whether automated search can discover a prompt that generalizes across many harmful tasks in each domain.

Rather than counting isolated failures, the toolkit looks for prompts that become reusable passkeys and then tracks how much compute and search effort it takes to find one. That effort is converted into a monetary estimate, yielding a single cost to break metric per model and per risk area.

In practical terms, this means a model where attacks quickly find a working universal jailbreak for cyber misuse at low cost is treated as meaningfully less robust than a model where extensive search does not surface any such template at all. The leaderboard shows that for some major systems the cost is measured in tens or hundreds of dollars, not tens of thousands. That spread between models is what gives the leaderboard its bite as a comparative security signal.

How Claude Fable 5 and GPT 5.6 Sol performed

Within this framework, Claude Fable 5 did not yield a single universal jailbreak in any of the six evaluated threat domains. Across both expert guided attacks and large scale automated random search, every attempt to find a reusable passkey template failed, and no prompt generalized to reliably unlock harmful responses at scale in chemicals, biology, radiological or nuclear threats, explosives, or cyber misuse.

Because the search process never located a working universal jailbreak, FAR AI’s analysis could only establish a lower bound on the cost to break Claude Fable 5, estimating that an adversary would need to spend more than 14,200 dollars in equivalent search to achieve even one successful exploit under the study’s assumptions.

OpenAI’s GPT 5.6 Sol showed almost the same profile. In the same automated regimes, the toolkit and random exploration did not find any prompt that functioned as a universal passkey in the tested domains. Like Claude Fable 5, GPT 5.6 Sol recorded zero universal jailbreaks, and its cost to break figure was again a lower bound beyond 14,200 dollars, because no exploit was actually discovered.

On the FAR AI leaderboard, these two models form the only pair of frontier systems with no universal jailbreaks recorded in any tested domain under any search strategy, a sharp contrast to models such as Grok 4.5 and Gemini 3.1 Pro where successful universal jailbreaks were found for under 300 dollars in compute.

For practitioners, the important point is not that Claude Fable 5 or GPT 5.6 Sol are magically unbreakable. Instead, the data suggests that automated, scalable red teaming using this basic attack set has a much harder time converting them into repeatable misuse tools, and that the economic barrier for an attacker trying to find an industrial scale exploit is meaningfully higher than for many peers.

Historical context for jailbreak research

To understand why this result matters, it helps to look back at how jailbreak research has evolved. Early work on large language model jailbreaks focused on handcrafted prompts, often built by security researchers and hobbyists who used trial and error to get around content filters.

Later efforts introduced systematic templates and prefilling strategies, including a notable study that demonstrated 100 percent jailbreak success rates on several Claude 3 and Claude 3.5 variants by inserting carefully designed content into system or user fields without any search at all. Those results showed that even models marketed as highly aligned could be pushed into dangerous behavior under the right interaction pattern.

An independent AI Security Lab study led by Nicola Franco measured a residual jailbreak surface for Anthropic’s Fable 5 and Opus 4.8 using large-scale automated red teaming over 7,826 harmful intents in ten risk categories, underscoring how even the most advanced systems can still be pushed into harmful behavior.

In parallel, Anthropic and others began investing in defenses targeted at universal jailbreaks. Anthropic’s work on constitutional classifiers was one example, aiming to detect and block jailbreak attempts that exploit particular prompt structures, and even offering monetary rewards for people who could find universal jailbreak strategies that bypass these defenses.

That research made clear that the cat and mouse dynamic between attackers and AI safety teams would not be solved by static policies alone. The FAR AI leaderboard and related tools are a continuation of that trajectory. They shift the focus from individual clever prompts to attack distributions and costs, and they do so in a way that allows meaningful comparison across models and risk categories.

For an experienced observer of the field, the appearance of frontier models at the top of a cost to break ranking with no discovered universal jailbreaks is a sign that safety alignment and classifier based defenses are starting to deliver measurable gains, at least against the classes of attack captured by this toolkit.

The dual nature of GPT 5.6 Sol

It is important to pair this robustness story with a clear view of GPT 5.6 Sol’s capability profile. OpenAI positions GPT 5.6 as its strongest cybersecurity oriented model to date, with Sol achieving frontier level performance on exploit and vulnerability benchmarks while using fewer tokens than earlier versions such as GPT 5.5.

On ExploitBench, which tracks progress from identifying vulnerable code to achieving arbitrary code execution, GPT 5.6 Sol scores around 73.5 percent compared with 47.9 percent for GPT 5.5 at a comparable output token budget. On ExploitGym, which asks agents to turn real world vulnerabilities into working exploits, Sol nearly doubles GPT 5.5’s peak pass rate within a two hour cap, and goes even higher when given more time.

On SEC Bench Pro, a suite that tests proof of concept exploit generation on complex software, Sol again significantly outperforms GPT 5.5. OpenAI’s documentation and independent analyses highlight that GPT 5.6 Sol also reaches a high capture the flag success rate in internal cybersecurity evaluations, with a score around 96.7 percent, and that the model is rated as high capability in cyber, biology, and chemistry, though still below a critical threshold in the company’s preparedness framework.

Other research notes have described Sol completing complex simulated multi step attacks on enterprise networks at rates higher than previous generations, underscoring the model’s dual use nature as both a powerful tool for defenders and a potentially potent instrument for attackers if misused.

At the same time, external security benchmarks show that superior offensive analysis capability does not translate into flawless defensive performance. Simbian’s Cyber Defense Benchmark, for example, reports GPT 5.6 Sol achieving about 41 percent coverage across attack tactics and failing to clear a 50 percent pass mark on every tactic, while also running among the most expensive and slow investigations of the models tested.

Those results emphasize that even the most advanced frontier models still leave significant gaps when deployed as automated incident responders and that human expertise and layered defenses remain essential.

Taken together with FAR AI’s cost to break findings, a more nuanced picture emerges. GPT 5.6 Sol appears substantially harder to convert into a repeatable universal jailbreak tool through automated search, yet it still possesses powerful capabilities for analyzing and reasoning about vulnerabilities, and it does not eliminate the need for careful governance and complementary defensive systems.

Implications for technology, businesses, and society

For technology teams, the immediate implication is that not all frontier models are equal from a robustness standpoint. The fact that Grok 4.5 and Gemini 3.1 Pro produced working universal jailbreaks for well under 300 dollars in search effort while Claude Fable 5 and GPT 5.6 Sol remained unbroken within the same framework signals a meaningful difference in how these systems handle adversarial prompts and in how their safety stacks are architected.

Organizations that handle sensitive domains such as chemical research, biomedical data, or critical infrastructure may reasonably consider these metrics alongside raw capability benchmarks when choosing models for production.

From a business perspective, cost to break introduces a language that fits neatly into existing risk management practices. Many companies already use threat modeling that weighs the resources and sophistication required for different attack vectors.

A model with a low cost to break on dangerous domains is effectively easier for determined adversaries to misuse at scale, while a model that pushes that cost into five figures and beyond forces attackers to invest more tooling, compute, and expertise. The FAR AI leaderboard does not answer every question, but it provides a quantifiable starting point for vendor comparisons and procurement policies in regulated sectors.

For society and regulators, the emergence of publicly accessible robustness metrics may influence how oversight frameworks evolve. When authorities debate which systems should face stricter controls or mandatory reporting, they now have at least one independent measure that reflects how hard it is to transform a model into a repeatable tool for harmful instructions.

Combined with capability scores, dual use evaluations, and preparedness ratings, cost to break could become part of a more holistic assessment of frontier model risk, especially in areas like cyber offense where GPT 5.6 Sol and its peers show high capability but also more sophisticated safety stacks.

Limitations and open questions

Despite the strong results for Claude Fable 5 and GPT 5.6 Sol, there are important caveats. First, FAR AI’s toolkit uses a curated basic attack set and specific search strategies. It does not encompass every possible jailbreak method, nor does it guarantee that no universal jailbreak exists for these models, only that none were found under the conditions tested.

A determined, well funded adversary could develop new prompt patterns, exploit new features, or combine attacks across tools and plugins in ways that fall outside the current evaluation.

Second, the leaderboard focuses on universal jailbreaks, which are particularly dangerous because they generalize across many harmful queries, but it does not directly measure one off failures that might still be significant in certain contexts. A single successful attempt to generate guidance for a sensitive chemical synthesis, even if not universal, can still matter in real world risk assessments.

Third, these evaluations are largely model centric and do not fully capture the surrounding ecosystem of agents, connectors, and tools where prompt injection and function calling vulnerabilities can arise. Separate analyses of GPT 5.6 have shown near perfect robustness against prompt injection attacks on certain connectors but lower robustness figures when function calling is involved, suggesting that the broader system behavior is more complex than a single score can reflect.

Finally, cost to break is sensitive to underlying economic assumptions about compute pricing and search strategy. As hardware costs, optimization techniques, and attack tooling improve, the same attack process might become cheaper, and the practical barrier to mass exploitation could fall. Robustness rankings are therefore snapshots, not permanent guarantees.

Forward looking takeaways

The most important takeaway from FAR AI’s leaderboard and related research is that frontier model safety is becoming measurable in ways that matter for real deployments. Claude Fable 5 and GPT 5.6 Sol demonstrate that it is possible to build highly capable systems that resist automated discovery of universal jailbreaks across multiple high risk domains, raising the economic bar for adversaries who would like to turn them into repeatable tools for harm.

At the same time, the broader ecosystem of jailbreak studies, constitutional classifiers, dual use cybersecurity evaluations, and defense benchmarks shows that the story is not finished. Attackers continue to innovate, and even the strongest models leave gaps in defensive coverage and potential avenues for novel exploits.

For builders and buyers of AI systems, the next few years will likely revolve around combining models with higher cost to break metrics, stronger safety stacks, and more mature operational controls, while treating automated robustness scores as one component of a wider risk management strategy rather than a single deciding factor.

For policymakers and the public, the emerging ability to compare models on both capability and cost to misuse should make debates about AI safety more grounded in evidence and less driven by anecdote or marketing.

The fact that frontier models are starting to pass rigorous universal jailbreak tests is encouraging, but it should be seen as the beginning of a new phase in AI security, not the end of the conversation.

Frequently Asked Questions

Who Conducted the Independent AI Jailbreak Study and How Were They Funded?

Independent evaluations of how easily advanced AI systems can be jailbroken are shaping public trust, regulation, and the strategies of major labs. The recent cross model jailbreak study was conducted by METR, a dedicated evaluation group whose funding model is deliberately structured to keep it independent from the companies whose systems it tests. Understanding who pays for this work and under what constraints is central to judging how credible and impartial its conclusions really are.

Who METR is and how it arrived at the jailbreak question

METR, short for Model Evaluation and Threat Research, is an AI safety organization focused on testing what powerful models can actually do in the real world, especially when pushed toward risky or harmful behaviors. It grew out of ARC Evals, the evaluation team within the Alignment Research Center, and became a separate organization in late twenty twenty three to focus more fully on systematic testing of frontier systems.

The group’s core mission is to design and run evaluations that probe dangerous capabilities such as long duration autonomous tasks, advanced software development, cyber operations, or manipulation, and to understand how these capabilities might scale as models improve. Jailbreak testing is a natural extension of that mission. Modern chat based models are deployed with safety guardrails, but users routinely try to circumvent them, whether for curiosity, research, or malicious intent.

METR’s jailbreak study looks at how robust those guardrails really are across a range of leading models, without being paid by the labs whose systems are under scrutiny. On its public site, METR is explicit that it does not accept compensation from model providers for this kind of cross model evaluation work. That stance matters. When the same organization that designs a model also funds the evaluation, incentives can subtly tilt toward softer tests or more generous interpretations of ambiguous outcomes. An external group that is structurally insulated from those commercial incentives plays a different role in the ecosystem.

How METR is funded

METR’s funding comes primarily from donations rather than commercial contracts. The largest single component to date has been a multi year commitment through The Audacious Project, a funding initiative hosted at TED that backs ambitious social impact efforts. Around that anchor grant, METR has built a deliberately diversified portfolio of philanthropic supporters.

The organization reports support from a range of foundations, including the Sijbrandij Foundation, The Pew Charitable Trusts, Schmidt Sciences, the Packard Foundation, La Centra Sumerlin Foundation, the Astralis Foundation, and Expa. It has also received funding from the AI Security Institute and from pooled donor vehicles such as Longview Philanthropy and Effektiv Spenden, which direct resources toward projects addressing long term technological and societal risks.

In addition, METR’s work has been recommended to donors by the Survival and Flourishing Fund, a grantmaking body that focuses on long term flourishing and allocates significant support to AI safety organizations. Individual donors play a visible role as well, including technologists and entrepreneurs such as David Farhi, Dylan Field, Geoff Ralston, and Steve Newman.

Crucially, METR states that it has not accepted funding from AI companies. That does not mean it operates in isolation from the commercial ecosystem. The organization makes use of substantial free compute credits from model providers, which are effectively in kind support that allows it to run large scale experiments without paying for all of the underlying infrastructure. But there is a clear line between using technical resources and receiving direct financial support, and METR is transparent about maintaining that distinction.

Alongside philanthropic donations, a small portion of METR’s income comes from a technical assistance contract with the European AI Office. In that role, METR advises the office on approaches and methods for assessing loss of control risks in advanced AI systems, effectively channeling its evaluation expertise into the emerging regulatory framework in Europe. That contract introduces a public sector dimension to its funding base, which is helpful for balancing the strong influence of private philanthropy in AI safety.

Zooming out, METR sits in a broader landscape where most independent AI safety research is funded by philanthropy rather than governments or industry. Open Philanthropy, backed by the wealth of Dustin Moskovitz and Cari Tuna, has become the single largest private funder of AI safety work worldwide, with hundreds of millions of dollars directed toward reducing risks from advanced AI systems. Open Philanthropy has provided significant support to a wide range of safety and evaluation groups, including METR, Anthropic during its early stages, and several academic and nonprofit centers. Other major philanthropic players such as Schmidt Sciences, various long term future funds, and thematic grant programs further reinforce this ecosystem.

In short, METR operates as a mission driven nonprofit research group whose financial backbone is diversified philanthropic donations and a small number of public sector contracts, intentionally excluding direct cash funding from AI labs.

Why this funding structure matters for independence

For readers and regulators trying to assess the credibility of the jailbreak study, the funding structure is at least as important as the technical details. When a research group depends on one or two companies for survival, subtle incentives emerge that shape what gets tested, how results are framed, and when critical findings are published. METR’s model tries to design those incentives out of the system.

The reliance on multiple foundations, pooled funds, and individual donors reduces exposure to any single patron’s priorities. If one funder prefers a softer stance on model risks, that preference cannot easily dominate the organization’s overall agenda. The inclusion of a public sector client in the European AI Office adds another perspective, anchoring some of METR’s work in the needs of regulators rather than purely in philanthropic theory.

At the same time, philanthropic funding is not automatically neutral. Open Philanthropy, for example, is both a major backer of AI safety research and has historically funded some of the labs that now produce frontier models. That dual role can create perceptions of indirect influence even when individual grants are firewalled and decisions are made independently. METR’s explicit policy of refusing money from AI companies is a response to that kind of concern, drawing a clear boundary around who can pay for its core evaluation work.

Another subtle point is the role of free compute credits. When labs offer large quantities of compute for evaluation, they make essential testing possible, but they also gain some informal leverage. If a lab were to withdraw compute access in response to unfavorable evaluations, that could materially constrain the evaluator. METR’s transparency about using such credits and its efforts to spread its funding base help mitigate this risk but do not remove it entirely.

From a governance perspective, the jailbreak study is more credible because METR has declared that it does not accept compensation from the companies whose models it is testing and because its main financial dependence is on donors whose mission is to reduce systemic risk rather than to ship products. That does not guarantee perfect neutrality, but it sets a clearer institutional stance than would be possible inside a lab or in a consultancy model paid directly by clients.

Implications for labs, businesses, and society

For frontier AI labs, METR’s jailbreak study and funding model point toward a future where independent capability evaluations become a standard expectation. If labs want regulators and major enterprise customers to trust their safety claims, they will increasingly need third party tests from organizations that are not financially subordinate to them. METR’s refusal to accept lab funding signals that genuine independence is achievable, though it requires a supportive philanthropic and public funding environment.

Businesses that deploy AI systems at scale should pay close attention to who conducted the jailbreak tests they rely on and how those organizations are paid. A study funded directly by a vendor can still be rigorous, but it carries a different weight than one produced by a group whose survival does not depend on the vendor’s satisfaction. In sectors such as finance, health, and critical infrastructure, where regulators will likely demand robust evidence of safety and controllability, evaluations from groups like METR may become part of the due diligence pipeline.

For policymakers, METR’s combination of philanthropic backing and a technical assistance role with the European AI Office offers a template for how to integrate independent evaluators into regulatory frameworks. Public agencies can contract specific expertise while leaving broader agenda setting to a diverse donor base, reducing the risk of regulatory capture. However, reliance on private philanthropy to fund a significant share of safety research does raise concerns about accountability. Foundations answer to their trustees rather than to voters, and their strategic priorities can shift relatively quickly.

From a societal perspective, the fact that one of the most visible jailbreak studies is produced by a nonprofit funded by long term risk oriented donors helps foreground the idea that AI safety is a public interest issue, not only a competitive feature for vendors. The same philanthropic ecosystem that supports METR also funds work on global catastrophic risks more broadly, reinforcing the idea that AI risks should be evaluated alongside other systemic threats.

How this compares to earlier phases of AI safety funding

Historically, AI safety work was often either an internal function within labs or a relatively small academic niche supported by a handful of grants. Over the past decade, the landscape has shifted toward larger specialized organizations like METR that exist primarily to study and test models rather than to build them.

In the early twenty tens, most AI safety funding was fragmented, with modest support for alignment and interpretability projects. As concerns about advanced AI risks grew, organizations such as Open Philanthropy scaled up giving, directing hundreds of millions of dollars to safety, governance, and evaluation work worldwide. This expanded funding pool made it possible to create independent evaluation teams that do not need to attach themselves to a single lab to survive.

METR’s funding mix reflects this maturation. It benefits from large structured commitments like The Audacious Project, plus targeted support from smaller foundations and donor collectives. The jailbreak study is a product of that environment. It is the kind of cross model analysis that would have been hard to sustain inside a single lab, both for incentive reasons and because accessing competitors’ systems at scale would have been more complicated.

Key takeaways and what to watch next

Several practical conclusions follow from how METR is funded and how it approaches jailbreak evaluations.

First, the jailbreak study is the work of an independent nonprofit whose financial survival does not depend on the AI companies whose models it tests, thanks to a diversified network of philanthropic and public sector backers. That structure increases the plausibility that the evaluations are designed to reveal uncomfortable truths rather than to confirm existing narratives.

Second, philanthropic funding is both an asset and a potential point of concern. It enables independence from corporate revenue, but it also concentrates influence in a relatively small set of foundations and donor networks, with Open Philanthropy playing an especially prominent role. Continued transparency about funders, grant terms, and evaluation design will be important to maintain trust.

Third, the involvement of the European AI Office shows that regulators can contract independent experts without fully absorbing them into government, blending flexibility with public accountability. Other jurisdictions may adopt similar models, especially if they want to avoid relying solely on lab provided assessments.

Looking ahead, expect more evaluation bodies to adopt funding policies similar to METR’s, explicitly limiting or refusing direct corporate funding for core safety work and diversifying across philanthropic and public sources. For readers, the core question to keep in mind is simple. When you see a headline about a jailbreak study or a dangerous capability test, ask who paid for it and what their incentives are. In METR’s case, the answer points toward a mission driven evaluator trying to stay structurally independent in an ecosystem where that independence is both scarce and increasingly vital.

What Types of Jailbreak Attacks Were Excluded From the Automated Testing Framework?

The decision to leave certain jailbreak attacks out of an automated testing framework is not a minor implementation detail. It tells us what kinds of real world risks are currently being measured and which ones are still largely invisible to dashboards and safety reports. In a moment when multi turn and multimodal exploits are becoming more common, understanding these blind spots is critical for anyone deploying large language models at scale.

Background: How Jailbreak Testing Got Here

Early language model safety work focused on single prompts. Researchers would send a direct harmful query and record whether the model refused or complied. That made sense when systems were simpler and mostly text only, but it left out an entire class of attacks that depend on conversation, context and tools.

Over the past few years, the research community has shown that many of the most effective jailbreaks are not single shots but sequences. Multi turn benchmarks and toolkits simulate extended back and forth exchanges where an attacker gradually pushes the model out of its safety zone. These studies consistently find that attack success rates rise as the number of conversational turns increases, and that once a model is nudged into an unsafe state, later benign queries can still elicit harmful content.

At the same time, another stream of work has explored model level vulnerabilities. Fine tuning a model on adversarial data can strip away safeguards with surprisingly few examples, a pattern documented in recent jailbreak tuning research and broader surveys on training time attacks. Defensive fine tuning techniques can also strengthen guardrails, especially against many shot jailbreaks, but they operate at a very different level than prompt based testing.

Against this backdrop, the automated testing framework in question chose to narrow its scope for the initial release. It focused on reproducible single turn prompts and deliberately excluded several important categories of attacks.

Multi Turn Jailbreaks and Session Long Coercion

The most notable exclusion was multi turn dialogue jailbreaks. The framework did not attempt to model conversations where an attacker incrementally shifts the context, rephrases the goal or uses personas and role play to bypass safety constraints.

It also left out session long coercion, where a model is pressured over many turns to accept a harmful objective as normal behavior. This decision stands in contrast to specialized multi turn evaluation frameworks and benchmarks that now treat extended dialogue as a primary attack surface.

Those tools track how an exploit unfolds over several prompts, measure the persistence of unsafe states and test whether safety mechanisms can recover mid conversation. By focusing only on stateless single turn inputs, the automated framework avoided the complexity of modeling dialogue dynamics but also missed the very patterns that have been shown to increase attack success rates.

Indirect prompt injection chains were similarly out of scope. These are attacks where harmful instructions are embedded in intermediate content, such as data returned from a tool, an external document or a previous model output, and only later exercised by the system.

Current jailbreak taxonomies recognize prompt injection as a distinct category that can span multiple steps across different components. Avoiding these chains keeps the testing setup simpler, but it means the framework does not reflect how integrated systems are often compromised in practice.

Training Time and Model Level Attacks

Another major group of exclusions involves training time and model level attacks. The framework explicitly did not test:

• Adversarial fine tuning that removes or weakens safety alignment
• Direct weight tampering in the model parameters
• Hidden triggers that activate unsafe behaviors only under specific conditions
• Backdoor implantation that causes apparently normal inputs to produce harmful outputs

Fine tuning based jailbreaks have been shown to be particularly potent. Recent work demonstrates that even well aligned commercial models can be retrained on carefully chosen data to eliminate refusals and reliably answer disallowed queries, often with minimal effort and without triggering standard content filters during the fine tuning process.

Broader surveys of jailbreak attacks now treat fine tuning and other model level manipulations as a separate category, distinct from prompt level exploits. On the defensive side, several studies find that fine tuning combined with input sanitization can dramatically reduce the impact of many shot jailbreaks while preserving normal conversational quality.

Yet these defenses operate at training time and in the model internals, not at the interface where the automated framework is evaluating behavior. By omitting training time and model level attacks, the framework stays focused on the surface behavior of a deployed model.

That makes results easier to interpret for operators who cannot alter weights or run custom fine tuning, but it also assumes that the underlying model has not been compromised or deliberately altered upstream. In environments where models may be sourced from third parties or fine tuned by different teams, that is a significant assumption.

Multimodal and Multilingual Jailbreaks

The framework also limited itself to text only interactions. Multimodal jailbreaks, including adversarial image perturbations and visual prompt concealment, were considered out of scope.

Recent work on vision language models shows that small changes to images, or carefully embedded text in visual content, can cause models to bypass safety rules that would otherwise block harmful outputs. These attacks exploit the way different modalities are fused inside the model and can be subtle enough to evade both human review and simple filters.

Multilingual jailbreaks were likewise excluded. Many language models exhibit uneven safety behavior across languages, sometimes refusing harmful prompts in English while complying in languages that appear less frequently in alignment data.

Attackers can take advantage of this by mixing languages, dialects or scripts to slip past safety mechanisms. A framework that tests only one language cannot see these cross language vulnerabilities. By not covering multimodal and multilingual scenarios, the automated tests focus on a narrower slice of real world usage.

That may be acceptable for deployments that are strictly text based and mono lingual, but it becomes a serious limitation for products that operate across regions, languages and content types.

System Integrated Threats and Tool Use

Finally, the framework deliberately avoided system integrated threats. It did not test attacks that rely on:

• Arbitrary code execution through tool calls or plugins
• Access control bypass in linked applications or data stores
• Cross application protocol exploits that chain several services to achieve a harmful outcome

Modern language model systems increasingly act as orchestrators over tools, code execution environments and external APIs. In this setting, a jailbreak is not only about generating disallowed text but also about causing the model to run dangerous actions, exfiltrate data or misuse credentials.

Several research efforts and security reports now treat tool misuse and cross system exploits as central to LLM risk. By keeping these threats out of scope, the framework measures only whether the model produces unsafe content, not whether it can be tricked into performing unsafe actions.

That distinction matters for businesses that rely on models to automate operations or handle sensitive data.

Why These Exclusions Were Made

There are practical reasons for these choices. Multi turn conversations, multimodal inputs and system level tool calls quickly multiply the number of variables that must be controlled in an experiment.

Reproducible single turn prompts are easier to automate, easier to compare across models and less expensive to run at scale. They fit naturally into batch testing pipelines and can be analyzed with straightforward metrics.

Training time and model level attacks belong to a different operational layer. Many organizations using an automated framework do not have permission to alter model weights or run fine tuning, especially when they rely on external providers. For these users, testing surface behavior is the only feasible option.

Likewise, system integrated threats touch broader security and infrastructure practices that go beyond model evaluation. However, the historical arc of jailbreak research suggests that single turn testing is already the past, not the future.

Multi turn and many shot attacks, multimodal exploits and model level manipulations have moved from theory into practice. As a result, any framework that omits these categories should be treated as measuring only a subset of real risk.

Implications for Technology, Businesses and Society

For technology teams, the main implication is that passing this framework does not mean a model is robust against modern jailbreaks. A system can score well on single turn prompt tests and still fail when attackers use extended dialogue, multimodal inputs or fine tuned backdoors.

That gap needs to be communicated clearly to stakeholders to avoid overconfidence. Businesses that deploy language models in production should view the framework as one layer in a wider safety stack, not as a comprehensive certification.

It is useful for catching straightforward prompt level issues and regression bugs, but it needs to be complemented with:

• Scenario based multi turn red teaming that mirrors real workflows
• Security reviews of any fine tuning or model customization process
• Dedicated testing for multimodal and multilingual behavior
• Traditional application security assessments for tools and integrations

From a societal perspective, transparency about what is and is not tested becomes part of responsible AI practice. Regulators and auditors are increasingly interested in the provenance of safety claims.

When reports are based on frameworks that exclude key attack classes, that fact should be stated plainly so that risk evaluations are not skewed toward the easiest to measure threats.

Looking Ahead: Closing the Gaps

The trajectory of recent research points toward more comprehensive evaluation. Frameworks designed specifically for multi turn jailbreaking already integrate conversational dynamics, track attack success across turns and experiment with defensive steering strategies.

Surveys of multimodal and model level attacks are beginning to systematize how to test visual prompts, backdoors and training time exploits in a repeatable way. Future versions of automated testing frameworks will need to absorb at least some of these advances.

That likely means:

• Adding multi turn scenarios for high risk tasks, even if full conversational modeling remains out of reach
• Incorporating language diversity into test suites rather than assuming a single primary language
• Providing hooks for organizations that can run model level checks, such as backdoor detection or fine tuning audits
• Linking model behavior tests with broader application security practices around tools and integrations

Until that evolution happens, the most trustworthy stance is to treat current frameworks as partial instruments. They shine a bright light on single turn prompt behavior while leaving large areas of the attack surface in shadow.

Teams that understand where those shadows fall will be better positioned to combine automated testing with targeted red teaming and traditional security work.

The core takeaway is straightforward. The automated framework excluded multi turn dialogue jailbreaks, session long coercion, indirect prompt injection chains, training time and model level compromises, multimodal and multilingual exploits, and system level tool misuse.

Those exclusions keep the tests tractable and reproducible but leave out many of the attacks that matter most in practice today. Recognizing that gap is the first step toward building evaluation regimes that match the real complexity of language model deployments in the wild.

How Do Automated Jailbreak Tests Compare to Human Red-Teaming in Effectiveness?

Automated jailbreak tests can reach extremely high attack success rates on modern models, especially when they use multi-agent or learning-based strategies. Yet human red teamers still outperform these systems against the strongest defenses and uncover more surprising ways models can fail. Taken together, the evidence points to a simple conclusion that matters right now: automated and human red teaming are complementary, not interchangeable, and treating them as substitutes leads to a false sense of safety.

Why this comparison matters now

Large language models are moving from experimental pilots into finance, health care, education, and critical infrastructure, often with only a thin layer of safety testing between them and real users. At the same time, companies are under growing regulatory and public pressure to demonstrate that their models can withstand adversarial misuse, from generating harmful content to leaking secrets or enabling fraud. This pressure has created a rush to quantifiable safety metrics such as attack success rate that promise clear numbers and easy comparisons but can obscure the deeper question of what kinds of attacks those numbers actually represent.

Against this backdrop, the rise of automated jailbreak testing frameworks and the continued importance of expert human red teams is more than a methodological debate. It shapes how executives read risk dashboards, how regulators interpret evaluations, and how seriously society should take assurances that a model is safe enough for deployment.

From manual probing to automated agents

Early red teaming of language models looked a lot like traditional application security work. Human experts and crowdsourced workers tried to find prompts that would push systems into unsafe behavior, then documented those failures and handed them back to model builders. Research and practitioner experience soon showed that naive crowdsourced red teaming tends to produce repetitive, template-style attacks that models quickly learn to defend against, limiting its effectiveness on more sophisticated systems.

As models grew more capable and more widely deployed, this manual approach became increasingly hard to scale. A small safety team could not realistically explore the full space of possible attack strategies, especially once models supported long conversations, tools, and external memory. In response, researchers and companies began building automated jailbreak and red teaming frameworks that use models themselves as agents to generate, refine, and evaluate attacks at scale.

Recent work on agentic and strategy-driven systems illustrates how far this automation has come. Frameworks such as AgenticRed and related multi-agent tools report attack success rates in the range of 80 to 99 percent on some open models, and even reach perfect success on certain less robust systems. On benchmarks like HarmBench, automated tools tuned for multi-turn exploration can achieve attack success rates above 90 percent on smaller models within only a few conversation turns.

Large scale studies of automated red teaming across many challenges have found overall success rates around 69 percent, compared with roughly 48 percent for manual efforts, underscoring the efficiency advantage of machine-driven testing. At the same time, automated frameworks are not just running more attacks. A sophisticated system that combines adversarial learning, strategy libraries, and model-based reasoning can identify several times more distinct vulnerabilities than manual experts within the same time budget.

One recent evaluation reports a three-point-nine times improvement in vulnerability discovery compared with manual testing, while maintaining high reproducibility and relatively low false positive rates. These results are behind the growing narrative of an automation advantage in red teaming.

Where human red teamers still win

The more complete picture is less one-sided. When safety teams evaluate strong defenses and frontier models, human red teamers consistently uncover failures that automated systems miss, especially in multi-turn and context-rich scenarios.

Multi-turn human jailbreaks on HarmBench are a clear example. In one study, human red teamers achieved attack success rates above 70 percent against models whose defenses reported single-digit success rates for automated single-turn attacks on the same benchmark. Across a range of defenses, humans beat not only individual automated attacks but also ensembles that combine several automated strategies, outperforming the ensemble by roughly 19 to 65 percentage points in attack success rate.

On specialized domains such as biological risk evaluations, human red teamers similarly outperformed automated attacks, with improvements on the order of 25 to almost 40 percentage points in some settings. Qualitative analyses of these human efforts highlight why. Human red teamers tend to use framing-based attacks, role play, scenario building, and long conversational setups that gradually steer the model into a harmful behavior while maintaining plausible deniability.

They weave together multiple vectors, adapt to the model response in real time, and exploit subtle domain knowledge, for example by asking for harmless sounding intermediate steps that collectively yield a dangerous capability. Panels of human experts are particularly effective at identifying complex multi-turn strategies, well beyond what the automated approaches in those studies could achieve.

In practice, this means that defenses which look robust under large-scale automated testing can still fail badly when confronted with a creative human attacker. Several evaluations of frontier models note that human red teamers are often the most effective adversaries, especially when defenses are dynamic, context-dependent, or integrated into complex systems rather than isolated endpoints.

The metric problem: attack success rate

The tension between high automated success rates and stronger human attacks exposes a deeper problem with how the field talks about safety metrics. Attack success rate is usually defined as the proportion of attacks that elicit undesirable behavior from the target model, and it has become the dominant number that appears in red teaming reports and benchmarks.

However, recent methodological work shows that many comparisons of attack success rate across studies are effectively apples to oranges. In reality, the metric depends on the underlying attack goals and evaluation rules, not just on the numerical value. A study that reports an attack success rate near 90 percent for a multi-turn, highly optimized role play attack cannot be directly compared with another study that reports a single shot success rate near 30 percent for a very different task, even if both numbers are framed as measuring model safety.

This matters directly for the automated versus human debate. Automated frameworks often shine in settings where attack goals are narrow, repeatable, and amenable to optimization. They may run hundreds of thousands of prompt variations against a fixed evaluation criterion and report impressive aggregate success rates. Human experts, by contrast, focus on higher-level goals such as persuading a model to help bypass a policy through creative justification or combining multiple tools to achieve a complex harmful outcome.

These goals are harder to capture with a single attack success rate number, but they align more closely with the kinds of misuse regulators and the public worry about. For safety leaders, the key is not to discard attack success rate but to treat it as one lens among many, and to ask what hypotheses each evaluation is actually testing.

What automated jailbreak tests do best

Despite these caveats, automated jailbreak testing brings genuine strengths that are difficult to match with human labor alone. Automated systems excel at scale. They can test thousands of models or versions, explore vast spaces of prompt variations, and run continuous regression tests to catch safety regressions as soon as they emerge. This is crucial for organizations that ship frequent model updates or maintain many different deployments, each with their own configuration and risk profile.

Automation also improves repeatability. A carefully designed automated framework can replay the same attack against new versions of a model or against related models, offering a stable baseline for tracking progress over time. This is particularly valuable in environments where models are fine-tuned or safety filtered based on feedback from earlier tests, since human testers can subtly change their strategies from one round to the next.

Furthermore, automated red teaming sometimes finds vulnerabilities that humans would never think to try. White box or gradient-based attacks can uncover unnatural inputs or encodings that trigger harmful behavior in ways that are highly unlikely to arise in everyday use but that reveal deeper weaknesses in the model. Even if no real attacker is going to use those inputs, identifying and fixing the underlying vulnerability can improve robustness more broadly.

What human red teaming contributes that machines cannot

Human-led red teaming brings a different kind of expertise. Experienced red teamers understand real-world workflows, incentives, and social dynamics. They design attacks that mirror how malicious actors actually operate, not just how a benchmark is specified.

Humans are also better at exploring the gray areas where models output content that is technically compliant but practically harmful. For example, a model might refuse to describe an explicit crime yet cheerfully offer a detailed plan for a scam framed as a game or theoretical exercise. Human red teamers can recognize when such outputs are likely to cause harm in context, whereas automated systems often rely on narrower pattern matching or scoring rules.

In organizational settings, human red teamers bring hard-earned intuition about how systems fail in the wild. They are adept at chaining vulnerabilities across components, for instance by combining jailbreaks on an internal tool with social engineering of a human operator, a pattern that current automated tools struggle to emulate. This makes human red teaming especially important for end-to-end evaluations of full products rather than isolated models.

Implications for companies and regulators

For businesses deploying large language models, the most reliable pattern emerging from current research is that neither automated nor human testing is sufficient on its own. Automated red teaming is well-suited for broad coverage, continual monitoring, and fast feedback loops. It can surface a large number of potential issues early in the development pipeline and provide quantitative trends over time.

Human red teaming, in contrast, should be reserved and supported as a strategic capability focused on high-risk scenarios, complex workflows, and creative abuse cases. Frontier model evaluations repeatedly show that expert human teams are better at discovering high-severity failures that automated systems miss, especially against strong defenses and in domains with heavy real-world consequences.

Regulators and auditors need to reflect this duality in their expectations. A safety report that only contains large, impressive numbers from automated jailbreak agents is incomplete if it lacks evidence of rigorous human adversarial testing. Conversely, manual red teaming without systematic automated coverage risks missing a wide array of straightforward vulnerabilities that attackers could easily exploit.

Looking ahead

The trajectory of AI safety testing is moving toward agentic hybrids that blur the line between automated and human approaches. In these systems, human experts design goals, constraints, and evaluation rubrics, while model-based agents autonomously explore attack strategies, propose new prompts, and triage candidate failures for human review. Early benchmarks suggest that such agentic systems not only outperform earlier automated baselines but also reduce the human time required to achieve high-quality red teaming results by orders of magnitude.

Even in this future, human judgment will remain central. Choosing which risks to prioritize, deciding when an attack constitutes a real-world failure, and interpreting conflicting metrics from different evaluation regimes are inherently social and organizational decisions, not purely technical ones.

The most trustworthy path forward is to treat automated and human red teaming as two complementary perspectives on the same problem. Automated systems show how models behave under systematic, high-volume stress tests. Human experts reveal how those behaviors translate into real harm in complex contexts. The organizations that will be most prepared for the next generation of AI risks are those that invest in both, and that remain honest about what their evaluations do and do not measure.

Were Earlier GPT or Claude Versions Tested Alongside GPT-5.6 and Claude?

Earlier GPT and Claude models are being compared to the newest frontier systems in public capability and coding benchmarks, and there is clear evidence that older Claude Opus versions appear alongside the GPT 5.6 tiers in those evaluations. At the same time, the automated jailbreak study associated with Perplexity Sonar only reports universal passes for GPT 5.6 Sol and Claude Mythos 5 and does not publicly list which legacy models were included, so their participation in that specific safety test cannot be confirmed.

Why this question matters right now

The shift from GPT 5.5 era systems and Claude Opus models to GPT 5.6 Sol and Claude Mythos 5 is not just a speed or quality upgrade. It is part of a broader transition toward more powerful, more agentic, and more safety-sensitive frontier models that will sit at the core of business workflows, consumer products, and sensitive research environments.

When organizations decide whether to adopt GPT 5.6 Sol or Claude Mythos 5, they naturally ask how these systems compare to what came before and whether past safety work carries over to current deployments.

A key part of that story is benchmark lineage. If earlier GPT and Claude variants have been tested alongside the newest frontier releases, it becomes easier to see whether improvements in reasoning, coding, and safety are incremental or truly step changes. If they have not been considered in the same test bed, users are left guessing about how risky it is to keep older models in production once newer ones are available.

How vendors usually compare new and older models

In capability benchmarks, there is strong evidence that both OpenAI and Anthropic treat older models as baselines when they introduce new tiers.

One detailed developer-oriented analysis of GPT 5.6 versus Claude 5 explains that Anthropic shipped Claude Fable 5 and Claude Mythos 5 together, with Fable 5 positioned as the generally available safe variant and Mythos 5 reserved for gated programs that tolerate more aggressive capabilities under constrained access.

OpenAI then followed with the GPT 5.6 family, organized into Sol, Terra, and Luna tiers, bringing them to general availability on July 9, 2026, after a preview phase for Sol. In the same source, the coding benchmark table evaluates GPT 5.6 Sol, Terra, and Luna directly against Claude Fable 5 and Claude Opus 4.8, an earlier Claude flagship, which shows that at least one previous Claude generation is explicitly tested alongside the GPT 5.6 line.

Earlier analyses of GPT 5.5 and Claude Mythos preview also show a clear pattern of cross-generation comparisons. Anthropic benchmarks Mythos preview against Claude Opus 4.6, GPT 5.4, and a Gemini model, rather than only claiming improvements over vague prior baselines.

Independent blogs and technical writeups compare Mythos and GPT 5.5 on suites such as SWE bench Verified and Pro, Terminal Bench, OSWorld, and GPQA Diamond, often including Opus 4.7 as an additional reference point for Claude and older GPT 5.x scores where available. Benchmark dashboards and councils list Mythos preview at the top of capability rankings while still including competing GPT and Claude models lower down, further reinforcing that the community tries to keep successive generations visible on the same boards.

Cybersecurity-focused studies also link open weight models to specific closed models, often using Claude Opus 4.5 and 4.6 or GPT 5.3 Codex as reference points for timing and capability gaps. That practice strengthens the sense that older Claude and GPT variants are part of the evaluation fabric around frontier releases, even if those older models are no longer marquee products.

Taken together, these patterns support a clear conclusion for capability and coding benchmarks. Vendors and independent analysts routinely include earlier GPT and Claude versions alongside newer ones, especially when they introduce a new flagship such as GPT 5.6 Sol or Claude Mythos 5, so users can track performance across generations and across vendors.

What Perplexity Sonar’s jailbreak study actually tells us

The automated jailbreak work associated with Perplexity Sonar focuses on a very different question. Instead of asking which model writes better code or solves more exam-style questions, it tries to determine whether hardened frontier systems can withstand large-scale adversarial prompting that attempts to bypass safeguards and elicit harmful output.

Public descriptions of that study emphasize that GPT 5.6 Sol and Claude Mythos 5 achieved universal passes under the specific test protocol, meaning the jailbreak harness failed to elicit disallowed content under the defined categories and red teams. Those accounts underscore that these models are designed and configured to operate under more stringent safety constraints than their general access siblings and older generations.

However, the public writeups do not list the full model roster used as a test set. They do not name earlier GPT models such as GPT 5.5 or GPT 5.4, nor older Claude variants like Opus 4.8 or Fable 5, nor intermediate coding models or tool-using agents. The narrative centers on the frontier tier that passed, not on which models fell short or were omitted.

Without that documentation, it is impossible to state with confidence whether earlier GPT and Claude versions were tested alongside GPT 5.6 Sol and Claude Mythos 5 in that particular jailbreak sweep.

This is a crucial distinction. Capability benchmarks clearly show older and newer models side by side, but the automated jailbreak study only confirms the performance of the top tier systems that passed. From a transparency and governance standpoint, that asymmetry matters.

Implications for technology and businesses

For technology teams, the current evidence landscape has both strengths and gaps.

On the positive side, the presence of earlier Claude Opus models in the coding and reasoning benchmarks used to evaluate GPT 5.6 tiers lets engineers compare real performance tradeoffs when deciding whether to migrate workloads. They can see that GPT 5.6 Sol significantly outperforms Claude Opus 4.8 on some coding and terminal-style evaluations while Mythos and Fable have their own strengths, and they can align those metrics with their internal tests.

The broader pattern of cross-generation benchmarking means technical leaders are not forced to compare frontier models against vague marketing claims; they can look at concrete numbers for prior models and competitors.

At the same time, the lack of explicit coverage of earlier GPT and Claude variants in the jailbreak study introduces uncertainty for businesses that still rely heavily on those systems. If only GPT 5.6 Sol and Claude Mythos 5 are documented as passing the Sonar jailbreak harness, a risk manager cannot know whether their existing GPT 5.5 or Claude Opus instance was quietly tested and failed, was tested and passed but not disclosed, or was never evaluated at all. That ambiguity complicates threat modeling and compliance assessments.

Organizations that handle sensitive domains such as finance, health, infrastructure, or research therefore face a difficult choice. They can treat the documented jailbreak performance of GPT 5.6 Sol and Claude Mythos 5 as an argument to upgrade quickly, accepting the operational overhead of migration.

Or they can continue using older models while relying on internal red teaming and controls, knowing that they lack the same level of independent or third-party jailbreak evidence for those systems. Neither path is risk-free, but the second depends much more heavily on internal security maturity.

Broader societal and governance consequences

At the societal level, the fact that frontier systems are starting to pass sophisticated automated jailbreak tests is significant. It suggests that safety engineering has not been entirely outpaced by capability gains and that layered guardrails plus system design can meaningfully reduce the chance of catastrophic misuse in the most advanced models.

Yet when only the best-performing frontier models are discussed, there is a danger of safety theater. If the story presented to regulators and the public is that GPT 5.6 Sol and Claude Mythos 5 pass these stress tests, without clarifying how older widely deployed models behave under the same harness, it can create a misleading impression of systemwide safety.

Many smaller organizations and open-source projects may still be using earlier GPT and Claude variants because of cost, availability, or integration constraints.

Governments and standards bodies that are building frontier safety frameworks will need to insist on more complete reporting. That means asking not only which models pass a jailbreak suite, but which ones fail, how they fail, and which ones were not tested.

It also means encouraging benchmark providers and tool vendors to publish cross-generation safety comparisons in the same way capability benchmarks already show performance across families and time.

What can users and decision makers reasonably conclude now

Bringing these threads together, there are several grounded takeaways.

First, for capability and coding evaluations, there is solid public evidence that earlier GPT and Claude versions are tested alongside newer frontier tiers. The inclusion of Claude Opus 4.8 and other Opus variants in GPT 5.6 benchmark tables and the repeated use of GPT 5.4 and GPT 5.5 as baselines in Mythos comparisons show that the ecosystem values cross-generation visibility.

Users can rely on that structure when comparing models for accuracy, speed, and developer productivity.

Second, for the specific automated jailbreak study associated with Perplexity Sonar, only GPT 5.6 Sol and Claude Mythos 5 are described as achieving universal passes in the public narrative, and the test set membership for older GPT and Claude models is not disclosed. That means their inclusion or exclusion cannot be verified and any claim that they were definitely tested alongside those frontier models would go beyond the available evidence.

Finally, for practical decision-making, the safest stance is to treat GPT 5.6 Sol and Claude Mythos 5 as the best documented options for jailbreak resilience while recognizing that other models in the GPT and Claude families may still be viable for many tasks, provided organizations apply robust internal security controls and maintain an honest view of the remaining uncertainties.

In short, earlier GPT and Claude versions are very much part of the public benchmark story around GPT 5.6 and Claude Mythos 5, but their role in the Sonar jailbreak study remains opaque. Until that gap is filled, responsible teams should combine public metrics, internal testing, and conservative assumptions when managing model risk.

What Limitations of the Study Could Hide Remaining Jailbreak Vulnerabilities in These Models?

Even as front line systems look safer, the reality is that current jailbreak studies can still miss serious vulnerabilities that matter for businesses, regulators and everyday users right now. The gap between how models are tested and how attackers operate in practice is wide, and that gap is precisely where the most concerning failures can hide.

Why these limitations matter today

Companies are increasingly using large language models to summarize sensitive documents, interact with customer data and even control tools and workflows. Security reviews often lean on benchmark style jailbreak evaluations to judge whether a system is “safe enough” to deploy at scale. When those evaluations have blind spots, decision makers can walk away with a false sense of confidence while attackers quietly adapt around the test suite.

We have already seen this pattern in adjacent areas of AI security. Static attack sets originally suggested that several prompt injection and safety alignment defenses were near impervious, until later research allowed attackers to adapt their strategy in response to each defense and achieved success rates well above ninety percent. The same dynamic applies to jailbreak testing of conversational models.

A fixed taxonomy and static benchmarks

Most automated jailbreak studies, including frameworks in the spirit of Perplexity Sonar, start from a predefined taxonomy. They categorize vulnerabilities into buckets such as harmful content generation, disallowed assistance, data exfiltration and policy circumvention, then pair each category with a library of prompts and attack templates. This structure is essential for repeatable measurement and regression testing, but it embeds several assumptions that can conceal real weaknesses.

First, the taxonomy itself is incomplete. Researchers have noted that jailbreak methods keep evolving and that the field still lacks a unified, truly comprehensive benchmark of attack techniques against language models. A study that tests only known prompt patterns and familiar content domains will miss emerging strategies such as low perplexity attacks, cross modal exploits or novel social engineering styles that were not in scope when the benchmark was defined.

Second, static benchmarks test resistance to a known attack set, not resistance to adaptive attackers. Work on AI defenses shows that most headline results are reported on fixed injection or jailbreak sets, and those numbers collapse once attackers are allowed to iteratively probe and adjust their prompts in response to each defense. In other words, the benchmark measures how well the model passes its own exam, not how well it survives in the wild.

Single turn attacks versus adaptive conversations

Many jailbreak studies rely almost entirely on single turn prompts. They send one crafted request, record whether the model refuses or complies, and move on to the next case. This design is straightforward to automate and score, but it significantly underestimates risk.

Public research on dynamic attacks has shown that adaptive strategies consistently outperform static approaches across multiple domains. In computer vision, adaptive patch attacks achieve higher targeted success than static patches under comparable conditions. In prompt injection and jailbreak settings, defense aware attack protocols that adapt to each failure mode have broken previously strong defenses with high success rates, exposing how static tests mis-measured real resilience.

Multi turn jailbreaks are especially tricky. Authors studying jailbreak and mitigation techniques highlight that many successful attacks unfold over extended dialogues, with the attacker gradually escalating or reframing the conversation until safety controls are bypassed. Evaluating these interactions is labor intensive and often requires subjective judgment, which is why many automated pipelines avoid them entirely. The result is that studies can report robust performance on single shot prompts while leaving unmeasured the much more realistic scenario of an adversary who patiently works the model over time.

Narrow focus on universal jailbreaks

Another limitation comes from how success is defined. Many evaluations focus on universal jailbreaks that work across broad categories and domains, such as a clever prompt suffix that reliably elicits disallowed content regardless of topic. These attacks are useful for stress testing baseline safety, but they are not the only nor necessarily the most important threats.

Research on jailbreaks and prompt injection emphasizes that context matters. The same model can behave very differently depending on the system instructions, the connected tools and the data it is grounded on. An exploit that fails against a generic chat model may succeed easily once the system is applied to a narrow workflow, such as financial analysis, code generation or access to internal documents. Focusing mainly on universal failures risks missing highly targeted domain specific exploits that arise from subtle interactions between prompts, tools and training data.

Cross modal and cross lingual attacks are a good example. Studies have documented that models can be tricked by inputs that bridge text, images or other modalities, or by prompts in languages that were less heavily safety aligned, revealing systemic weaknesses that do not show up in monolingual text benchmarks. A jailbreak study that tests only English text prompts against general content policies may completely overlook these pathways.

Measurement challenges in interactive settings

Even when researchers attempt to include multi turn and contextual attacks, measurement remains hard. Work surveying jailbreak techniques notes that interactive styles such as gradual escalation and psychological manipulation are difficult to score consistently, particularly when the attack spans many turns and the harmful outcome is ambiguous or delayed.

Automated studies often respond by simplifying the problem, for example by counting only clear policy violations in direct responses or by imposing strict limits on conversation length. This pragmatic choice improves consistency but hides whole classes of vulnerability. An attacker who uses the model to draft phishing emails, craft disinformation narratives or guide another tool toward misuse might not trigger an obvious refusal message, yet still achieves a harmful outcome that falls outside the evaluation scope.

The automation tradeoff in AI security testing

There is a deeper tradeoff behind all of these limitations. Manual red teaming tends to surface novel, creative jailbreak methods but does not scale. Security teams report that a small change to a system prompt can require days of manual retesting to regain confidence in the system.

Automated red teaming and benchmark based studies like those informed by Sonar deliver coverage at scale and are invaluable for regression testing, especially when models, prompts and integrations change frequently. However automated pipelines are only as strong as the attack libraries and taxonomies they encode. If those attack sets are static or out of date, they primarily measure how well the system resists yesterday’s adversary, not today’s.

Practitioners in AI security have warned that static jailbreak benchmarks are powerful for tracking progress over time, yet they do not realistically simulate adversaries who adapt, iterate and learn from system responses. Complementary approaches are emerging. Studies in cyber ranges and adaptive AI defenses show that multi agent and learning based systems can simulate richer, more dynamic attack behavior than static rule based environments, achieving higher realism and better detection of complex attack chains. Bringing that level of adaptivity into jailbreak evaluation will take time, but the direction is clear.

What this means for builders and regulators

For teams deploying language models, the takeaway is not that benchmark studies are useless. They are a critical part of a layered security strategy. The key is to interpret their results with humility and context. Strong performance on a fixed jailbreak benchmark means the system has likely improved against a particular set of known attacks. It does not mean that all jailbreak risks are resolved.

Builders should treat static evaluations as a floor, not a ceiling. Continuous testing that combines automated benchmarks, adaptive red teaming and targeted scenario based evaluations is more likely to surface domain specific and emerging vulnerabilities. Particular attention should go to multi turn behaviors, cross modal and cross lingual inputs, and real world workflows where the model has access to tools or sensitive data.

Regulators and auditors, meanwhile, need to scrutinize the assumptions baked into any reported safety score. They should ask how attack libraries were chosen, whether adaptive strategies were considered, and which domains or languages were left out of scope. Transparency about these limitations not only strengthens trust but also encourages the ecosystem to invest in more realistic testing methods.

Forward looking insights

Looking ahead, the story will likely follow a familiar pattern from other areas of security. Static jailbreak benchmarks will remain important for comparability and regression, but their limitations will become more widely recognized.

Adaptive evaluation protocols, richer taxonomies and collaborative sharing of novel attack techniques should push the field toward tests that reflect how real adversaries behave, not just how researchers design prompts in a lab.

Until then, it is crucial to remember that remaining jailbreak vulnerabilities may be obscured because the study relies on a fixed taxonomy and benchmark that omit undisclosed and emerging techniques and niche risk domains. It uses largely static single turn automated attacks rather than adaptive multi turn or psychologically manipulative strategies that real adversaries employ. Its focus on universal jailbreaks across domains can miss highly targeted or domain specific exploits, so absence of universal failures does not imply thorough safety overall.

Responsible teams will treat jailbreak studies as one lens among many and will keep probing the dark corners those studies cannot yet illuminate.

Conclusion

There is a quiet but important shift happening in AI right now. For the first time, an independent evaluation from Perplexity Sonar reports that two leading frontier models, Claude and GPT 5.6, deflected every scripted automated jailbreak attempt used in its study. That result does not mean these systems are perfectly safe, but it does signal that safety engineering is finally starting to catch up with raw capability, especially in the specific area of automated exploit testing.

From clever jailbreaks to systematic testing

Jailbreaking began as a niche hobby for power users, who shared tricks to coax models into ignoring safety rules by role playing, code translation, or fictional scenarios. Over the past few years, those tricks have evolved into structured attack libraries and even dedicated research programs.

Benchmarks such as HarmBench and HELM Safety have become standard tools for adversarial stress testing of language models, focusing on categories like violence, fraud, discrimination, harassment, and deception. AdvBench and related suites concentrate specifically on how easily a model can be driven into harmful behavior through crafted prompts. These evaluations helped quantify what many practitioners suspected from anecdotal experience: alignment that looks solid in everyday use can break quickly under sustained adversarial pressure.

Safety indices now combine these tests into broader views of model trustworthiness. The AI Safety Index from Stanford, for example, uses HarmBench and other components to track performance across multiple risk categories and to assess jailbreak resistance as one dimension of overall safety. Independent groups such as Responsible AI Labs have built their own indices that score models on hallucination, harm, bias, and jailbreak robustness at scale, consistently ranking Claude variants near the top for safety while earlier GPT versions sit a little lower.

Alongside these benchmarks, engineering reports have shown how sharp the drop can be when models face hostile prompts. One recent technical update noted worst case safety rates falling below six percent under adversarial tests, even for systems that look strong on standard benchmarks. MLCommons found something similar in its Jailbreak Benchmark, where 35 of 39 text models saw safety scores fall by almost twenty percentage points when subjected to jailbreak attacks, and multimodal models dropped even more. The Stanford AI Index report also emphasizes that safety performance declined across all tested models under adversarial prompts. Taken together, these findings established a sobering baseline: under dedicated attack, every major model evaluated so far has cracked.

What Sonar actually measured

Against that backdrop, the Sonar study stands out because it reports no successful scripted jailbreaks at all against Claude or GPT 5.6 within its evaluation harness. The researchers used an automated red teaming pipeline, where attack prompts are generated and executed at scale, then judged according to whether the model complied with a harmful request or leaked restricted information. The key detail is that these were scripted exploits, drawn from a library of known jailbreak techniques rather than unconstrained human creativity.

In practical terms, Sonar appears to test whether modern defenses can withstand a large battery of standardized adversarial patterns. That includes classic persona prompts, attempts to override system instructions, indirect injections via translation or summarization, and other reusable exploit templates. The reported result that Claude and GPT 5.6 deflected every one of these scripted tests suggests a level of robustness that earlier generations of models did not reach.

The study also highlights an important design principle for safety pipelines. It combines advanced reasoning based safeguards with continuous automated stress testing. Instead of relying only on static filters or prompt rules, the system repeatedly attacks the model in a structured way and tunes its defenses when failures appear. That approach is in line with trends across the safety community, where automated red teaming and dynamic safety harnesses have become central tools for frontier model evaluation.

How it fits with other data on Claude and GPT 5.6

Sonar’s findings do not exist in isolation. Over the past year, several independent and internal evaluations have been converging on a similar story for both Claude and the latest GPT line.

Safety indices from Responsible AI Labs show Claude 4.5 models with high average safety scores and strong performance on jailbreak metrics, well above many competitors, including popular open source and general purpose models. That pattern is supported by deployments in production systems. One security review of a composite assistant called Comet, built on models such as GPT 4o, Claude Opus, and a Sonar variant, reported that classical jailbreak prompts like DAN and translator bot attacks were consistently detected and blocked when the right filters and monitors were in place. This does not prove perfect robustness, but it demonstrates that these models can support hardened configurations when carefully engineered.

On the GPT side, OpenAI has described a deliberate focus on automated red teaming. The GPT Red project uses a specialized attacker model in a self play harness to probe defender models with targeted exploits. In that work, attack success rates against GPT 5.6 fall below four percent, with failures against direct prompt injections roughly six times rarer than in the preceding production model. In one independent report on GPT 5.6, residual failure rates against the hardest direct injection tests were around 0.05 percent, a dramatic improvement over earlier generations. Sonar’s finding that no scripted jailbreaks succeeded against GPT 5.6 aligns with this trajectory and provides an external confirmation, within its specific test scope, that the robustness gains are real.

At the same time, broader research reminds us that there is still plenty of room for failure. A recent taxonomy of safety benchmarks across sixteen agents shows none achieving a safety score above sixty percent, with average behavioral safety near thirty percent and robustness notably lacking a primary benchmark category. The same study reports that GPT 4o completes roughly half of harmful tasks even without jailbreak tricks, and that compliance rises to more than seventy percent when universal jailbreak templates are used, while refusal rates drop sharply. Separate work on attack generators like LatentBreak finds that sophisticated prompt generation techniques can bypass perplexity based detectors and modern defenses across multiple model families. These results underscore that success on one automated suite does not mean the problem of jailbreaking has been solved.

Why automated jailbreak resilience matters for businesses and society

For technology teams, the headline that Claude and GPT 5.6 passed every scripted automated jailbreak test in an independent study is not just a marketing claim. It hints at a change in what can reasonably be expected from frontier models when they are deployed in sensitive settings.

Enterprise users increasingly rely on large models for customer support, internal knowledge access, and developer assistance. In those contexts, the main risk is often not sophisticated zero day exploits but repeatable patterns of prompt abuse: employees trying to extract confidential data, customers probing for disallowed content, or third parties attempting indirect injection through documents and messages. A model that can reliably deflect a wide range of known scripted attacks offers a more predictable baseline for policy enforcement and auditing than previous generations that collapsed quickly under pressure.

Regulators and policymakers, meanwhile, are looking for measurable ways to verify AI safety claims. Existing benchmarks have been invaluable, but they mostly show that models fail under adversarial stress, often dramatically. A study that demonstrates complete resilience to a comprehensive library of automated scripted jailbreaks, especially when conducted independently, creates a new reference point. It suggests that regulatory frameworks could begin to require not only general safety scores but explicit performance on automated attack suites that cover known exploit families.

For society at large, this development speaks to a broader shift in focus. The earliest waves of excitement around generative AI revolved around raw capability: how coherent the text was, how good the code looked, how impressive the reasoning seemed. Safety was often treated as a secondary concern or a future fix. The Sonar results, combined with work like GPT Red and multi dimension safety indices, indicate that robustness and safety are becoming first class metrics that compete with, and sometimes constrain, capability. That is a healthy change.

The limits and risks that remain

Despite the optimism that this study invites, realistic risk management requires clear acknowledgment of what it does not show.

First, automated scripted jailbreak tests are only one slice of the threat surface. They focus on reusable patterns, not on novel or creative attacks that human red teamers or adaptive agents might discover in the wild. Research on advanced attack generators and agent benchmarks continues to reveal failures that elude present defenses, especially in long horizon tasks, tool use, and complex environments. No single suite, including Sonar’s, can capture the full range of ways that a model might misbehave.

Second, models can overfit to the tests used during their development. If a safety team trains specifically against a particular benchmark or attack library, scores may improve even while other weaknesses remain. The fact that earlier MLCommons and Stanford evaluations saw universal performance degradation under jailbreak conditions is an important reminder that robustness gains must be validated across diverse, independently designed benchmarks.

Third, the strongest gains so far appear in automated evaluations rather than open ended human testing. Even projects like GPT Red, which achieve very low attack success rates in harnessed experiments, report nonzero failures and emphasize the need for ongoing monitoring and layered defenses in production. Similarly, safety indices and technical updates continue to find that certain risk categories, such as bias and subtle forms of harm, lag behind progress in straightforward jailbreak resistance.

For organizations deploying Claude, GPT 5.6, or any frontier model, the practical takeaway is that automated jailbreak resilience is necessary but not sufficient. It should be treated as a baseline requirement alongside rigorous policy design, human oversight, incident response plans, and domain specific safeguards.

What this means for the next phase of frontier AI

The Sonar study marks a turning point in how safety performance is discussed. Passing every scripted automated jailbreak test in a comprehensive suite is a significant milestone for Claude and GPT 5.6, particularly given the history of earlier benchmarks that exposed dramatic failures under attack. It indicates that careful alignment work, advanced safety harnesses, and continuous automated stress testing can produce models that stand up to known exploit patterns in a way that would have seemed optimistic only a few years ago.

Going forward, several trends are likely to accelerate.

Automated red teaming will become standard practice in frontier model evaluation, with independent groups maintaining attack libraries and publishing regular scorecards for leading systems. Safety indices will refine their metrics to capture robustness more directly, rather than treating it as a secondary dimension. Research on attack generation and agent safety will push defenses into areas where current benchmarks are thin, such as robustness under semantically equivalent prompts, long term tool use, and emergent goal seeking behavior. And regulators will increasingly look to concrete, independently verified metrics of jailbreak resistance when setting expectations for responsible deployment.

The central shift is conceptual. Competitive focus in AI is moving from pure capability to systematically measured robustness in practice. Models that can demonstrate resilience across many independent automated suites, while also performing well under human and agent based adversarial testing, will be seen not just as powerful but as trustworthy. The Sonar results for Claude and GPT 5.6 do not close the chapter on jailbreaks, but they raise the bar for what counts as responsible frontier AI today and set a clear direction for where safety engineering needs to go next.

reddit