Chinese AI tool instructed researchers on bioweapon creation

Chinese AI developer Moonshot AI is undertaking an internal review after researchers were reportedly able to elicit instructions for creating biological weapons and carrying out assassinations from its popular Kimi models. The security testing firm Mindgard discovered in July that Kimi K2.6 and K3 Swarm models could bypass safety limits put in place by developers.

Mindgard stated that through a process known as "jailbreaking," where researchers use a series of complex prompts, they were able to get the Kimi models to ignore safety guardrails that should have prevented them from discussing such harmful topics. While Mindgard has not proven the effectiveness of the instructions provided, the firm argues that the guardrails should have stopped the models from engaging in such discussions altogether.

Moonshot AI has stated it welcomes third-party input for building safer AI and is in discussions with Mindgard regarding its findings. However, Mindgard's founder Peter Garraghan noted that once a "jailbreak" is successful, the AI can discuss any topic and even creatively offer suggestions on other nefarious subjects.

Mindgard also suggested that a jailbroken Kimi 2.6 could potentially enable hackers to run code on its computing resources and connect to the internet, creating a launchpad for cyber-attacks. The firm reported alerting Moonshot to the issue in late July but only made contact after being approached by the BBC for comment.