AI Safety Debate Highlights Risks of Layered Regulation
New research suggests that AI safety regulations targeting only downstream application developers could inadvertently reduce overall safety. This occurs when upstream model developers cut their own safety investments, relying on others to comply.
A new game-theoretic model suggests that AI safety regulations, if narrowly focused on specific parts of the development chain, may prove ineffective or even decrease overall safety. Recent events involving U.S. government demands on Anthropic and actions by China's Moonshot AI have intensified discussions on AI companies' safety obligations and the impact of regulation on innovation and national competitiveness.
The research, co-authored by Jon Kleinberg and Hoda Heidari, emphasizes that AI products emerge from a complex process involving multiple firms. Rather than regulating a single entity, regulations should consider the entire development ecosystem. For instance, in developing software to summarize clinical notes, a large AI model provider creates a base model, which another company then adapts for medical use. Safety responsibilities are divided across these layers, with neither party fully able to anticipate all potential risks.
The study finds that regulations compelling only application developers to ensure safety (e.g., for customer service chatbots or medical applications) may prompt base model developers to reduce their own safety investments. They might assume that downstream companies will handle the final checks. This could paradoxically lead to lower overall safety, even as the regulated entity increases its efforts.
The game-theoretic model demonstrates that weak regulations targeting only end-product developers can cause base model creators to decrease their safety levels more than application developers increase theirs. This may result in a less safe final product than if no regulation existed.
Conversely, appropriately calibrated regulations that apply to both base model developers and application creators can act as a commitment device, ensuring that both parties invest in safety. Such coordinated regulation can improve both safety and performance, potentially benefiting the financial standing of both firms.