Anthropic, OpenAI's proposed AI risk evaluators may not have enough power to prevent disasters

Anthropic CEO Dario Amodei wants independent evaluators embedded inside frontier AI companies, explicitly citing bank supervision as the precedent. Bank supervisors can compel action or close an institution, but to date it is not clear that the proposed AI watchdogs will be given a similar power. "If you don't give them that kind of power, I don't know what they're doing," said banking regulation expert Julie Andersen Hill. Another limitation, according to hands-on model evaluator Albert Ziegler at cybersecurity company XBOW, is that today's tests can expose important large language model failures but may never trigger the rare combination of behavior that produces a catastrophic outcome. Anthropic CEO Dario Amodei has proposed embedding third-party safety evaluators inside frontier AI companies on an ongoing basis as part of a plan to slow the advance of increasingly capable models, after former Anthropic researcher Jacob Coxon resigned, warning that frontier labs were racing toward systems they might not control. In an essay published last weekend that has led to a regulatory split between factions within the AI community and government, including President Trump, Amodei committed Anthropic to providing third-party evaluators with access comparable to internal risk teams, and the right to publish findings without the company's editorial control, subject to limited redactions. He explicitly cited embedded bank supervisors as a precedent, writing that his proposal "has precedent in the banking industry, which sometimes involves regulatory 'supervisors' embedded along with employees." But Julie Andersen Hill, dean of the University of Wyoming College of Law and expert on banking regulation, says without the power to flip the "kill switch" on the whole operation, the comparison to the banking industry regulation isn't accurate. "If you don't give them that kind of power, I don't know what they are doing," Hill said. At the largest banks, government examiners have offices inside the institution, access to internal systems and employees, and a continuous presence. They can direct a bank to stop a practice, restrict its growth, force management changes, and in extreme cases close it, Hill said. Anthropic's proposed evaluators would be able to investigate and report, but Amodei's plan does not give them comparable enforcement power or the legal authority to prevent a model from being trained or released. "That's fundamentally different, because bank regulators have a lot more power than that," Hill said. Amodei wrote that evaluators are needed to provide "a neutral third party who can actually see the details" but acknowledged that Anthropic still determines what it includes and omits from its existing public disclosures. Details released to date suggest the evaluators would have extraordinary access to frontier AI systems but limited formal authority over the companies developing them. Neither Anthropic's proposal nor OpenAI's existing third-party evaluation framework gives outside evaluators independent authority to halt the development or deployment of a model.