Mandatory capability tests for advanced AI models are moving from abstract policy talk to concrete proposals, and Anthropic is one of the companies trying to define what that new safety regime should look like. This matters now because frontier systems are crossing thresholds where they can meaningfully accelerate cyber operations, biological research, and automated AI development, creating risks that look less like individual misuse and more like systemic national security challenges. As policymakers absorb the potential for third-party testing to become a formal legal gate for deploying frontier systems, Anthropic’s push for capability-linked safeguards helps frame what those statutory obligations might entail.
How we got to mandatory capability testing
For most of the last decade, AI safety has been governed largely by voluntary commitments, ethics boards, and a mix of technical safeguards such as content filters and alignment training. Governments pushed some transparency and risk management, but they largely allowed labs to decide when a model was “safe enough” to deploy.
For years, AI safety relied on voluntary norms, ethics boards, and lightweight technical safeguards
That started to change as models became capable of assisting with tasks that go far beyond harmless productivity gains. Anthropic’s Responsible Scaling Policy introduced the idea of explicit capability thresholds and AI Safety Levels that tighten controls as models become more powerful in sensitive domains such as chemistry, biology, and autonomous AI research. The company has since iterated that policy, adding specialized thresholds for chemical, biological, radiological, and nuclear related capabilities and refining how those thresholds connect to required safeguards. Additionally, dedicated safety teams are crucial for managing the risks associated with advanced models.
In parallel, Anthropic developed a broader Advanced AI Framework that sketches how governments could impose mandatory obligations on the most capable models, especially those that might be used at scale in critical infrastructure or national security contexts. Together, these documents move the conversation from general concern about “frontier AI” to concrete criteria about which models should be regulated and how.
The core idea: pre-deployment certification for frontier AI
Anthropic’s policy agenda centers on a simple but consequential concept. Extremely capable AI models should not be widely deployed until they pass safety tests that specifically target catastrophic misuse risks. In public talks and documents, chief executive Dario Amodei compares this to aircraft certification.
Advanced models, in this view, should go through technical testing and auditing before they are allowed to operate at scale, and regulators should have authority to delay or reverse deployments when tests reveal unacceptable risks.
To make that workable, Anthropic argues that not every AI system needs this level of scrutiny. Instead, it proposes a focus on frontier systems and developers whose models are powerful enough, and widely used enough, to create systemic risk. That is where the concept of “covered models” and “covered developers” comes in.
Under the Advanced AI Framework, a developer is covered if it trains models that require more than (10^{25}) floating point operations and also either earns more than five hundred million dollars per year in AI derived revenue or spends more than one billion dollars annually on AI research and development. The floating point threshold is intended to capture frontier scale training runs, while the financial criteria limit the rules to a small number of large labs that are driving the most advanced systems.
On the model side, anything above roughly (10^{25}) training operations would trigger a mandatory pre-deployment review process, modeled after the kind of structured safety assessment familiar from aviation and nuclear regulation. Over time, Anthropic acknowledges that compute requirements for dangerous capabilities will fall, and suggests that governments may need to switch from a pure training cost threshold to direct capability based triggers.
Responsible Scaling Policy in practice
Anthropic’s Responsible Scaling Policy is where this testing regime becomes concrete inside a single company. The policy defines capability thresholds as prespecified levels of AI performance that, once crossed, require stronger safeguards than the current safety level provides. Each threshold is tied to an AI Safety Level, abbreviated ASL, that bundles specific deployment and security obligations.
The basic pattern is conditional. If a model achieves certain capabilities that matter for catastrophic risk, then the lab must activate stricter safeguards. For instance, when a model reaches the point where it can significantly assist in the development or use of chemical or biological weapons, the policy requires ASL 3 protections.
Those protections include tighter access controls, stronger security around model weights, and sophisticated input and output classifiers that block content associated with dangerous techniques and weaponization pathways. Anthropic describes ASL 3 as the level where a system substantially increases the risk of catastrophic misuse compared with non-AI baselines such as search engines or textbooks, or shows early forms of autonomous capability.
The company commits not only to perform red team testing at that level, but to withhold deployment if world class adversarial testing reveals any meaningful catastrophic misuse risk that cannot be mitigated with available safeguards. In practice, Anthropic reports that it activated ASL 3 protections for relevant models in mid 2025 and has been hardening those defenses since then.
The policy also includes an evaluation cadence. Anthropic designs its assessments to trigger at slightly lower capability levels than the ones it is most worried about, and evaluates its systems at defined intervals roughly every factor of four increase in effective compute. This is meant to create a safety buffer so that models do not overshoot dangerous capability thresholds in between evaluations.
Recent updates add more nuance. The company has introduced a CBRN development threshold targeted at capabilities that could substantially uplift the development capacity of moderately resourced state programs, and it has refined how it treats autonomous AI research capabilities, moving some earlier concerns into explicit checkpoints rather than automatic triggers for higher safety levels.
For cyber capabilities, Anthropic’s publicly available system card notes that the Responsible Scaling Policy does not yet stipulate a formal threshold at any ASL level, which reflects both measurement difficulty and ongoing debate about what counts as a dangerous uplift in offensive cyber capacity.
Why open weight frontier models are a special concern
A recurring theme in Anthropic’s proposals is that open weight models pose particular risks once they reach frontier capability levels. If a model can meaningfully help a technically trained user design biological agents or probe critical software systems, broad access to its weights makes it much easier for malicious actors to replicate, fine tune, and modify those capabilities in uncontrolled ways.
Because of this, the Advanced AI Framework suggests pairing mandatory testing obligations with stringent security expectations for any covered open weight model. These expectations include hardened infrastructure to resist sophisticated intrusion attempts, strong internal access controls, and careful monitoring of how weights are stored and shared.
Under ASL 3, security standards are explicitly elevated to address the risk that a non-state adversary could steal model weights and deploy them outside any safety regime. From a policy perspective, this is an attempt to square two competing pressures. On one side, governments and national security communities worry about the proliferation of models that can assist in weapons development or cyber offense.
On the other side, researchers and smaller companies emphasize the importance of open tools for innovation, competition, and scientific progress. Anthropic’s stance is that capability linked obligations for the highest risk open weight models can reduce the chance of catastrophic misuse while still leaving room for open development where risks are lower.
The testing pipeline Anthropic envisions
Mandatory capability tests have to be practical, not just aspirational. Anthropic therefore advocates for a testing pipeline that starts with fast automated screening and escalates into deeper expert-led evaluation when needed. Automated tests can probe whether a model crosses predefined performance thresholds in domains such as biology or chemistry, using standardized tasks and success metrics.
When those tests indicate that a model is approaching or exceeding a dangerous capability threshold, human experts step in to conduct more detailed red team exercises and scenario analyses. In the regulatory proposals, governments would require covered developers to run these tests before major deployments, publish safety frameworks and regular risk reports, and engage qualified independent evaluators who can scrutinize both results and methodology.
Regulatory authorities would need real enforcement tools including the ability to demand changes, delay releases, or block deployments that pose significant catastrophic risk. At the same time, Anthropic warns against broad rules that sweep in routine systems or impose heavy compliance burdens on small actors, arguing that regulation should be tightly scoped to models and developers whose capabilities and scale justify this level of oversight.
Implications for technology and business
If some version of Anthropic’s approach becomes law, the most immediate impact will fall on a small cluster of frontier labs. Training runs above (10^{25}) operations and budgets in the hundreds of millions of dollars are still concentrated among a handful of companies, so mandatory capability testing would largely be a requirement for them rather than for university labs or startups working on modest models.
Technically, this could change how frontier models are planned and released. When crossing a capability threshold triggers expensive security upgrades and the possibility of non-deployment, research teams may start to think much more deliberately about which capabilities they pursue and how quickly they scale them.
There is a risk that labs could try to game tests by shaping benchmarks or narrowing what counts as “meaningful assistance” in sensitive domains, which is why independent evaluators and transparent metrics are critical for trust. For businesses that rely on frontier models, mandatory testing and safety levels could provide clearer assurances about misuse risks and security practices.
Enterprises deciding whether to integrate a powerful model into financial systems or healthcare workflows would benefit from knowing that the model has passed structured tests and is subject to ongoing monitoring and emergency controls. On the flip side, increased testing and regulatory review could slow the cadence of major releases and make some capabilities less accessible or more expensive, at least in the short term.
Smaller companies and open source communities would be affected indirectly. As frontier labs tighten controls on high-risk capabilities, some research may shift into safer domains where tests are easier to pass. That could push cutting edge open source work toward areas like reasoning and tooling rather than chemistry or biology assistance.
Over time, however, as compute and algorithmic efficiency make powerful capabilities possible with fewer resources, thresholds based solely on training cost may become outdated. The Advanced AI Framework already anticipates this and suggests moving toward more direct capability criteria, but designing those criteria in a way that is robust and not easily manipulated will be challenging.
Societal and governance consequences
From a societal perspective, mandatory capability tests represent a shift from hoping that labs will “do the right thing” to requiring demonstrable evidence that high-risk systems meet defined safety standards. For national security agencies, this offers a more tangible handle on AI risk.
Instead of debating general fears about artificial intelligence, they can focus on specific capabilities such as assisting individuals with basic technical training in building or deploying chemical or biological weapons, and insist that any model crossing that line be subject to ASL 3 style protections.
The approach also raises important questions. How much authority should regulators have to block or roll back models that powerful companies have already trained? How transparent should evaluations be, especially when they involve classified threat scenarios or proprietary model internals? What happens when different countries adopt conflicting thresholds or standards?
Anthropic’s proposals provide a detailed starting point but they do not fully resolve these geopolitical and institutional challenges. There is also the issue of measurement. It is relatively straightforward to count training operations or revenue, but much harder to measure whether a model “significantly assists” in a complex real world task such as bioweapons development or state level cyber campaigns.
The Responsible Scaling Policy uses proxy tasks, automated benchmarks, and red team simulations, yet all of these depend on current knowledge and imagination, which can miss novel misuse pathways. A credible regime will need continuous improvement, external critique, and a willingness to update thresholds when evidence shows they are too lax or too strict.
Key takeaways and what to watch next
Mandatory capability tests for AI models mark an important evolution in how advanced systems are governed. Anthropic is arguing that frontier models should be treated more like aircraft or nuclear facilities, with structured pre-deployment safety checks and clear thresholds that trigger stronger protections.
The Responsible Scaling Policy and Advanced AI Framework together sketch a pathway for targeting regulation at the most capable systems and largest developers, aiming to reduce catastrophic misuse risk without smothering routine innovation.
Whether this vision becomes reality will depend on how governments translate it into law, how other labs respond, and how well capability thresholds track the changing frontier of what models can do. The next few years will likely bring more detailed public benchmarks for biological and chemical assistance, clearer criteria for cyber capabilities, and early examples of regulators either approving or halting high profile model launches.
For now, Anthropic’s work offers one of the most concrete blueprints for mandatory safety testing in AI, and it is a blueprint that developers, policymakers, and businesses would be wise to understand and critique in detail.
Conclusion
Artificial intelligence policy is moving from abstract principles to concrete requirements, and Anthropic’s new push for mandatory capability tests on high risk open weight models is a significant step in that shift. It matters right now because the frontier of open models is colliding with serious cybersecurity and biosecurity concerns, and governments are starting to ask what should be allowed on the internet in the first place, not just what big companies promise to do.
How frontier and open models arrived at this moment
For most of the past decade, openness in machine learning meant open source code, accessible research papers, and model checkpoints that anyone could download and fine tune. That culture helped drive rapid progress in areas such as computer vision and language understanding, and it made it much easier for startups and academic labs to compete with large incumbents.
The arrival of very capable general models changed the risk profile. Systems that can write working malware, accelerate biological design tasks, or help users bypass safety controls introduce the possibility of real world harm when they are widely available in an unrestricted form. Evaluations of modern models already show that even top tier systems exhibit unsafe behavior in a large fraction of tasks under realistic conditions, with failure rates approaching half of evaluated tasks in some categories. That is the backdrop for Anthropic’s argument that open weight releases need a different safety bar once models cross certain capability thresholds.
Regulators have started to move in the same direction. California’s emerging rules for frontier AI already require large developers to publish safety frameworks and assess whether their systems could materially contribute to catastrophic harms such as large scale casualties or major property damage. That kind of language, which would have seemed extreme only a few years ago, is becoming standard in discussions about the fastest growing models.
What Anthropic is actually proposing
Anthropic’s Advanced AI Framework lays out a concrete regulatory structure for the most capable models, including those that might be released with open weights. The company proposes that governments impose special obligations on developers that both train models above a very high compute threshold and meet substantial revenue or research spending thresholds, so that the rules focus on genuinely frontier scale actors rather than hobbyists or small labs.
For these covered developers, Anthropic wants mandatory testing of the most capable models against four specific risk categories. These are biological weapons assistance, offensive cyber operations, loss of control over AI systems, and automated research and development that could significantly accelerate dangerous capabilities. Under the proposal, developers would need to evaluate every qualifying model against each of these categories and publish summaries of the results, alongside a clear safety framework and detailed system cards that describe capabilities and risks.
The framework goes further than many existing disclosure rules by calling for regular risk reports at least twice a year that describe how a company’s overall risk posture is changing and what mitigations are in place. Anthropic also emphasizes that independent evaluators should review these assessments, with governments helping to set standards and fund an ecosystem of qualified third party testers who have meaningful access to the models.
Crucially, this is not meant to be a purely voluntary code. Anthropic explicitly argues that regulators should have enforcement power, including the ability to delay, block, or even reverse deployment of models that fail to meet high safety standards or that are found to pose significant catastrophic risk. In public remarks, Chief Executive Dario Amodei has compared this approach to aviation regulation, where aircraft must pass technical testing and auditing before they are allowed to carry passengers, and unsafe designs can be grounded in the interest of public safety.
All of this fits with the company’s internal Responsible Scaling Policy, which already ties release decisions to capability thresholds and escalating safety levels. That policy introduces AI Safety Levels that increase safeguards as models acquire more advanced capabilities, with stronger pre deployment testing, security hardening, and operational oversight kicking in once models approach high risk domains such as autonomous AI research or assistance with chemical and biological misuse. The new public framework is essentially an attempt to turn pieces of that internal regime into external rules for the whole industry.
Why open weight models are a special case
The debate becomes especially sharp when the models in question are open weight. Unlike services that run on a company’s servers, open weight models can be copied, modified, and redeployed by anyone once the weights are released. That makes standard safety tools such as content filters or hosting controls much less effective, because an attacker can strip them away or fine tune the model to ignore them.
Recent research on open weight models argues that this difference justifies a proportional evaluation regime that is specifically tailored to the unique risks of freely accessible weights. The authors propose that evaluations should be run on minimally wrapped models that are tested without safety prompts or filters, so that developers understand the true baseline capability and risk profile before release. They also recommend tampering tests that simulate malicious fine tuning or weight modifications, stress tests that look at targeted amplification of high risk capabilities such as cyber offense or biological design, and worst case misuse simulations that approximate what a determined actor with substantial compute could do.
Anthropic’s call for mandatory capability tests aligns naturally with this line of thinking. If a model can be copied and modified without meaningful oversight, then the decision to publish its weights looks more like exporting a dual use technology than launching a typical cloud service. Under that framing, requiring standardized tests for cyber, bio, alignment, and control risks before release is a way to treat open weight deployment as a regulated act rather than a routine software update.
How this could reshape the balance between openness and safety
The most important implication of Anthropic’s proposal is that safety becomes a precondition for openness instead of being framed as its opponent. By tying open weight deployment to clear, standardized evaluations that focus on specific harms, the company is arguing that robust testing can preserve the benefits of an open ecosystem while narrowing the window for catastrophic misuse.
Technically, this would push the open source community toward a more mature safety culture. Standardized tests for cyber and bio risks, and for loss of control scenarios, would give both developers and downstream users a shared vocabulary for discussing what a given model should and should not be trusted with. As evaluation methods improve and benchmark suites evolve, that shared language can become a foundation for more nuanced governance, such as gradually tightening thresholds or recognizing new risk categories when evidence supports them.
For businesses, the proposal cuts both ways. On the one hand, it would likely increase compliance burdens for companies that want to train and release very capable open weight models. They would need in house safety teams, relationships with independent evaluators, and the internal controls to generate defensible risk reports. On the other hand, a clear regulatory regime could reduce uncertainty for enterprises that want to build on open models, because they could rely on documented tests and government backed standards instead of marketing claims alone.
From the perspective of open source communities and smaller labs, the appeal depends on how the thresholds are set and enforced. Anthropic’s framework attempts to limit mandatory obligations to developers that cross both very high compute and financial thresholds, which would exclude most hobby projects and research groups that fine tune existing models. That design is meant to avoid smothering low risk innovation while still putting strong guardrails around systems with genuinely frontier level capabilities.
Regulators face a different trade off. Granting agencies the power to block or reverse model deployments raises obvious concerns about overreach, politicization, or regulatory capture that might entrench a handful of incumbents. On the other hand, without some enforcement mechanism, mandatory testing quickly becomes aspirational and the most aggressive actors can simply ignore it. The challenge will be designing processes that are transparent, appealable, and grounded in technical evidence rather than political pressure.
Relationship to existing policy trends
Anthropic’s proposal does not appear in a vacuum. It echoes and extends elements of existing and emerging regulation. California already requires frontier developers to publish detailed frameworks that assess catastrophic risks, report serious safety incidents to authorities, and keep those documents up to date. Anthropic’s suggestions would effectively generalize and harden these obligations for the most capable models, with stronger expectations for independent evaluation and clearer authority to halt dangerous deployments.
The company’s broader policy on the AI exponential lays out a vision where frontier developers are required to test their models, publish summaries, describe how they manage catastrophic risks, and engage with qualified independent evaluators who have adequate access to systems. It also stresses strong security around the entire development environment, including protections against insider threats and sophisticated external attacks, again with expectations of both public transparency and confidential reporting channels to designated agencies. Taken together, these ideas are building blocks for a regulated commons in which transparency and auditing are part of the basic social license to operate advanced AI systems.
The internal Responsible Scaling Policy reinforces this trajectory. It shows that Anthropic is willing to commit, at least on paper, to withholding deployment of models that cross certain capability thresholds if safety measures are not sufficient, especially in areas like autonomous AI research or assistance with chemical, biological, radiological, or nuclear misuse. Turning similar commitments into binding external rules would reduce the gap between what leading companies say they will do and what the entire ecosystem is actually required to do.
Open questions and real risks
Even with a clear conceptual framework, the details remain complicated. Defining which models count as high risk, and which capabilities should trigger mandatory tests or deployment blocks, is technically and politically contentious. Capability evaluations are still immature, and studies repeatedly find that models can behave far more dangerously under carefully chosen prompts or in realistic tool use settings than they do on static benchmarks. A test suite that looks robust today may miss the most important failure modes that appear tomorrow.
There is also a question of global coordination. If only a handful of jurisdictions mandate serious testing for high risk open weight models, developers might route training or release through more permissive countries. That risk is one reason Anthropic and other firms emphasize international standards and collaboration, including shared criteria for what counts as a catastrophic risk and common expectations for risk reporting and incident disclosure. Achieving that kind of alignment in practice is difficult, but without it the impact of any single country’s rules will be limited.
Finally, there is a cultural question within the open source world. Many researchers and engineers see openness as a core value in itself and worry that strict testing regimes will be used as a pretext to restrict access, stifle competition, or centralize power in a few large firms that can afford major compliance programs. Anthropic’s focus on targeting only the most capable systems is an attempt to address those concerns, but trust will depend on how thresholds are set, how often they move, and whether the rules are enforced consistently rather than selectively.
What to watch next
The clearest takeaway is that capability based regulation for open weight models is no longer a theoretical idea. A major frontier developer is actively urging lawmakers to adopt it, complete with concrete thresholds, defined risk categories, and a role for independent auditors with real teeth. That is a meaningful evolution from earlier debates that framed openness and safety as fundamentally opposed.
In the near term, two developments will be especially important. First, whether any national or state level law adopts something close to Anthropic’s proposal, including explicit authority to block deployments that fail capability tests. Second, whether evaluation science can keep pace, producing tests that reliably capture real world misuse potential rather than offering a false sense of security.
For practitioners, the practical implication is to assume that advanced open weight releases will increasingly be judged not just on benchmark scores but on documented safety evaluations, independent audits, and the developer’s broader risk management posture. Building models that are powerful, understandable, and controllable will become a competitive advantage, not just an ethical aspiration.
The long term question is whether this approach can create a genuinely regulated commons for AI, where openness is preserved for a wide range of beneficial uses, yet the most dangerous systems are constrained by credible, enforceable guardrails. If Anthropic’s vision or something like it takes hold, the story of open AI will be less about publishing everything by default and more about earning the right to release the most capable models through rigorous, transparent testing and shared accountability reddit
