Anthropic and OpenAI need truly independent safety evaluators, experts say in public letter

Anthropic and OpenAI need independent safety evaluators, experts say

Over 100 artificial intelligence experts and evaluators are banding together to warn they won't have the necessary resources and protections to test the safety of AI technology, which is facing heightened scrutiny due to concerns from insiders about the potential dangers of frontier models.

"We're just trying to really demonstrate a shared common ground on basic principles and ensure that independent oversight can be a meaningful tool for managing AI risk broadly," said Conrad Stosz, chair of the AI Evaluator Forum consortium that organized the letter, in an interview.

The group published a public letter on the matter on Friday, and shared it exclusively with CNBC. The signatories include AI luminaries like Geoffrey Hinton and members of organizations such as Johns Hopkins University, Stanford University and the nonprofit evaluator METR.

They want to compel foundation model providers to ensure that third-party AI evaluators are allowed the necessary "scientific objectivity, transparency, independence, and robust protections" to do their jobs effectively and credibly, the letter said.

Stosz said it's part of an effort to hold the foundation model companies accountable to their recent pledges to support more thorough third-party AI safety testing.

The niche community of evaluators has been catapulted into the limelight since Anthropic CEO Dario Amodei floated the idea over the weekend of providing some of them "employee-like access" to inspect and audit bleeding-edge foundation models and their development processes.

While some industry leaders have called on the government to regulate AI development to ensure it's not spinning out of control, President Donald Trump and his former AI czar, David Sacks, have adamantly opposed such efforts.

Stosz said that the coalition doesn't "advocate for one particular way" to ensure that AI models are developed safely, but wants to ensure that "basic principles" and "greater standardization" are at least established for evaluators and others who work independently of the major labs.

Amodei's proposal, Stosz said, appears to involve providing significantly more access than evaluators have previously enjoyed. Such a scenario, Stosz said, could involve foundation model makers giving third-party evaluators access to company computers, allowing them to talk to employees candidly and letting them "see sensitive internal data and unreleased systems."

"That type of access would give us much greater confidence and certainty about the actual risk, particularly for systems that they're using internally and not releasing," Stosz said. He cited the unreleased OpenAI model used in the Hugging Face attack.

OpenAI CEO Sam Altman, SpaceX's Elon Musk and Microsoft CEO Satya Nadella have publicly supported Amodei's proposal, but they've yet to address the logistical issues with such an undertaking, such as which AI evaluators will be selected and how deeply they would get to inspect...