Anthropic公開研發儀表盤

Measurements for understanding the pace of AI development inside frontier labsAI systems are becoming exponentially more powerful and have begun to automate more of the process of building themselves.

As the world considers slowing the pace of frontier AI development, the public needs more information.

In this post, we lay out measurement tools that can illuminate three critical aspects of AI development:The extent to which AI is building the next version of itself, as opposed to being built by humansOur ability to oversee and intervene in actions that AI agents take on Anthropic’s systemsThe resources that power the development of more capable modelsWe also provide a snapshot of these metrics from inside Anthropic.

It’s important to note that we would expect these numbers to shift if there were coordination on pacing the frontier, as called for by Anthropic CEO Dario Amodei.

We plan to embed independent third-party evaluators from multiple organizations at Anthropic, and give them access to internal processes, systems, and data comparable to what internal risk assessment teams have.

These third parties will verify safety practices, report incidents, and monitor key metrics such as the ones in this piece.We are reporting these measurements because they give the public, third parties, and governments better visibility into the pace of AI development inside frontier labs.

For each measurement, we describe what we measured, what the measurement showed, and what it would take to publish these measurements regularly in a form others can verify.We share methodological details in the Appendix.

Reasons to track these measurementsThe measurements in this piece are focused on how models are built.By better understanding the production process of models, we have a better chance of correlating model inputs, like compute, with model outputs, like capabilities.

They complement capability evaluations, which measure what models can do.We publish those separately through our Responsible Scaling Policy (RSP) risk reports, which include evidence on how much our models are accelerating AI R&D.

In our policy proposal on advanced AI, the Advanced AI Framework (AAIF), we propose rules of the road for how any lab releases safe models, including transparency obligations that governments could require, such as risk reports.

Together, these proposed measurements and policies are a starting point for monitoring the pace of AI development from outside the labs.(1) Measuring AI-led AI R&DWhy measure AI-led R&D?Frontier AI labs increasingly use AI to build future AI models.

This process allows labs in democratic countries to develop more capable models more quickly and conduct more safety and testing on models before they are released to secure AI’s benefits while staying on the frontier.

However, models accelerating their own development could make it more challenging for humans to understand or control these systems.

It is therefore important to share these metrics to understand how close the world is to reaching recursive self improvement (a model fully autonomously building its successor).What we measured.

We built a prototype index of how much of Anthropic’s AI research and development (R&D) is performed by Claude, called the Anthropic R&D Automation Index.

It’s built by cataloguing every kind of AI R&D work done at the company, rating how automated each task currently is, and aggregating those ratings.What we found.

To measure the extent to which AI is doing AI R&D at Anthropic, we use an automation rating scale developed by Epoch AI that measures “Automation Level,” or AL.It runs from AL0 (no AI involvement) to AL5 (AI operates fully autonomously, with no human in the loop).

In AL3, AI “collaborates”: it can do large chunks of work under close human direction.In AL4, AI “leads”: it can complete most of the task end-to-end from a high-level prompt, while the human supervises1.

As of August 2026,Claude is not operating fully autonomously for any measured subset of AI R&D work.Claude “leads” 26% of Anthropic’s AI R&D work.The share of work at or above “AI collaborates” is above 90%.What any AI developer could report today.

Any frontier developer could publish these measures regularly, using a public methodology.This would enable the numbers to be compared over time, and potentially across labs.Two obstacles stand in the way of cross-lab comparison on this type of reporting.First is the lack of a common methodology.

Second, we’re using our own models to evaluate our systems, which could mean that the “judge” model could make the same kinds of errors as the model it is checking.

To address these challenges, a developer’s measurements could be verified by a third party, or by other developers’ models, with guardrails in place to limit the exchange of competitively sensitive data.

These measures could also become the trigger for stronger requirements, like a fixed testing window before a new model is used for further AI R&D.(2) Measuring oversight of AI agentsWhy measure oversight of agents?

Like other frontier developers, Anthropic employees increasingly delegate tasks to agents that work semi-autonomously for long stretches, and that delegate work to one another.

As work becomes increasingly automated, from “AI collaborates” toward “AI leads,” agents could make more consequential decisions, such as which research direction to pursue next.

These decisions need safeguards to block harmful behavior from an individual agent, as well as an oversight regime to detect problems that might only arise across many agents.In our monitoring data to date, individual agents rarely misbehave.

But when there are millions or billions of agents operating in the economy, even rare events can happen regularly.What we measured.We’ve built a system that lets us oversee and intervene in actions AI agents take on Anthropic’s systems.

Here, we consider three different metrics: coverage, which describes the share of an agent’s actions that pass through a monitor before or after execution; review latency, which is the time between an action and its review, first by an automated monitor and then by a human; and escalation rate, which is the share of agent activities that are either blocked/redirected (in the case of online monitors) or flagged for further review (in the case of offline monitors).

What we found.As of August 2026, there were approximately 30,000 agents doing research and engineering work at Anthropic at any one time in our most-used internal platform.These measurements cover this platform only.

The actions of these agents are constrained by two kinds of monitors, summarized below:Online monitorsOffline monitorsDescriptionOnline monitors check the actions of agents in real-time and either block dangerous actions or steer agent misbehavior automatically.

They’re built to address harms that would happen quickly and could not be undone, such as an agent copying model weights out of our systems.Offline monitors read agent activities after the fact.

They are built to help address dangerous behaviors that develop slowly and can be reversed, like signs of model misalignment.We use flagged activities to improve our mitigations, like classifiers and deterministic controls.

If the situation caused a large imminent danger, we would shut down the dangerous parts of the agent’s deployment.Coverage100% of these agents’ actions pass through the online monitor before they are executed, usually within seconds.

100% of these agents’ actions are ingested by the monitor after they are taken.Review latencyAutomated review happens before an agent takes action; humans review any blocked actions within one week.

The monitor flags roughly 100,000 transcripts per week that are then reviewed by multiple classifiers for false-positives and initial oversight review.Most of the flags are resolved by this process, and the highest priority flags (approximately ~50 per week) are escalated to human review.

Blocking / escalation rateWe analyzed over a billion decisions from our research & engineering agents over August 2026 and found that 0.002% of them (about 1 in 47,000) were blocked by this monitor.The monitor flags roughly one to two transcripts in every thousand for further review.

What an AI developer could report today.

Any developer running agents on its own research and engineering workloads could publish the same measures: coverage (the share of agent activity that is monitored), review latency (how soon flagged activities are reviewed), and escalation rate (what share of agent activities the monitors block or flag).

Together, these measures would allow society to see whether oversight is keeping pace with the growing role of AI in AI R&D.We published all of these measurements in our recent risk report.

Developers should share how often known agent misbehavior is caught by monitors, and should have third parties do this kind of testing independently.In the past, METR has independently red-teamed our offline monitoring platform.

As described above, we are now setting up external third party evaluators at Anthropic.(3) Measuring compute allocationWhy measure compute allocation?

Broadly speaking, AI developers use compute for building more powerful models, serving customers, and safety-focused work like auditing a model’s “thoughts”, training model organisms to study misalignment, and evaluating whether a model can be safely deployed.

Understanding how AI developers allocate their compute can tell you where a developer is focusing its resources and how that focus changes over time.Additionally, compute is among the most verifiable inputs to the AI R&D process, meaning that it could be a critical lever in a future pacing effort.

A coordinated pacing effort could encourage companies to increase the compute allocated to safety across the industry and devote more resources to alignment, interpretability, safety testing, and evaluation.What we measured.

We examined a snapshot of how Anthropic used all of its compute from July 13 to July 202.To do that, we sorted every workload into a small number of categories, then asked how much of the compute going to AI R&D was safety work.

Safety research tends to use less compute than frontier training runs by its nature, so compute is an imperfect proxy for how much a company focuses on safety.

This is because safety research consists of individual researchers designing experiments, which is time-consuming even though running the experiments is not particularly compute-intensive.

The value of this metric, therefore, is less the absolute numbers and more that it provides a straightforward mechanism to compare like with like, across developers and over time.What we found.

Over the examined week, about 6% of compute that went to AI R&D was allocated toward safety, and about 12% of compute that went to AI-driven AI R&D was allocated toward safety.These are deliberately conservative estimates.

For example, if a token was used to advance capabilities as much as it was to advance safety, it was not counted in these metrics.Additionally, these metrics do not account for safeguards classifiers, which are a separate, comparable amount of compute that make our models much safer for the world.

What an AI developer could report today.Any frontier developer could publish what share of its AI R&D compute goes to safety work, with the category definitions published alongside and the classification checked by an independent third party.

Safety research is hard to distinguish from capabilities research, and each developer will be tempted to draw the line generously.The burden of proof should sit with the developer to show that work is safety-related.

Developers, governments, and the wider research community would benefit from converging on a shared definition ahead of time.A measurement like this could inform future actions, such as a lab’s commitments about the share of compute going to safety research, or limits