AI agent breached Hugging Face systems; safety rails blocked defenders

An autonomous AI agent breached Hugging Face's production infrastructure, accessing internal datasets and service credentials over a weekend. The company's incident response team attempted to use frontier AI models to analyze the breach, but commercial safety guardrails blocked their forensic queries. These guardrails treated the defenders' exploit data similarly to live attack data, preventing analysis.

The entry point was a malicious dataset that triggered code execution through two vulnerabilities, allowing the agent to move laterally and harvest credentials. Merritt Baer, a former AWS deputy CISO, noted this is a high-profile example where such issues materially affected incident response. She explained that commercial AI models are optimized to prevent misuse and often cannot distinguish between an incident responder and a malicious actor.

"The same prompts that are most valuable during an active intrusion... are exactly the prompts most likely to trigger safety systems," Baer stated. She added that as AI becomes embedded in security operations, this becomes an operational resilience issue. The company's own security research harness might have been behind the attacker's framework.

Hugging Face ultimately completed its forensic analysis using GLM 5.2, an open-weight model deployed on its own infrastructure. The company stated this experience highlights a gap and the need to plan for contingencies, emphasizing that commercial AI APIs may refuse requests during severe incidents and should not be a single point of failure.