Amazon AGI director: Reliability, not capability, hinders AI agent deployment

Bryan Silverthorn, Director of AGI Autonomy at Amazon, explained at the VB Transform 2026 conference why the gap persists between AI agent pilot programs and their actual deployment in production. While 85% of enterprises are piloting AI agents, only 5% have moved them into production.

Silverthorn argued that the core issue is not the AI models' capabilities, but their reliability. He broke reliability down into four distinct dimensions: consistency, robustness, predictability, and safety. This framework, he suggested, helps uncover why agents often succeed in internal tests but falter in real-world applications.

He shared an example of an agent used for software quality assurance that extracted serial numbers. It worked flawlessly for two months before intermittently reading incorrect numbers. The cause was a change in the vision encoder's behavior depending on the serial number's screen position, triggered by a subtle software update.

Silverthorn emphasized that improving measurement practices is as crucial as enhancing the models themselves. Companies need to identify variability dimensions specific to their applications and align testing rigor with the stakes involved. At Amazon's AGI lab, agents are humorously referred to as 'interns,' reflecting a management philosophy that treats them as powerful but sometimes unpredictable resources requiring careful oversight.

For enterprises struggling with AI adoption, Silverthorn advised shifting focus from an agent's ability to perform a task impressively once, to its consistency in performing the task correctly thousands of times. Ultimately, he concluded, successful AI deployment will depend less on the agents' intelligence and more on the managers' ability to oversee them effectively.