Anthropic backs court power to block risky AI models

Anthropic supports giving courts authority to suspend AI models that pose serious risks, according to a letter it sent to California Gov. Gavin Newsom, reports Politico. The proposal responds to California Gov. Gavin Newsom's September executive order on AI oversight and shutdown controls.

Anthropic said the state attorney general should be able to seek a court order when a model presents a near term risk of catastrophic harm. A court could suspend its deployment, internal use or further development. Large AI developers would also publish risk reports at least every six months.

The company supports requiring frontier AI developers to maintain systems that can quickly shut down models. It said monitoring should detect problems and allow companies to respond at the level where they occur, while taking an entire model offline should be reserved for extreme cases.

Anthropic also proposed placing independent safety evaluators inside AI companies with access comparable to internal risk staff. Developers would pay into a state administered fund rather than paying evaluators directly, while state officials could rate and assign the evaluators.

OpenAI sent Newsom a similar letter last week supporting shutdown controls and embedded auditors, but it did not go as far as backing government power to block a high-risk model.