Expedia's AI Lead: Evaluations Are the New Product Requirements

<h2>Evaluations Become Centerpiece of AI Product Development</h2> <p>At the VB Transform 2026 conference in Menlo Park last week, Xavi Amatriain, Expedia Group's chief AI and data officer, declared that "the new PRDs are the evals." He explained that evaluations, which may include methods like red teaming, will define what a product should do and incorporate security requirements before the coding even begins.</p> <p>Amatriain, who joined Expedia in December 2025 after serving at Google, believes that AI-assisted or AI-generated code will fundamentally alter future development processes. "All your thinking is gonna go into the evals," he stated. These insights gain significance when considering VentureBeat's Pulse research, which found that 66 percent of enterprises permit or are moving towards production deployment without human review, yet only 5 percent fully trust automated evaluations.</p> <h2>Balancing Flexibility and Security</h2> <p>Amatriain also cautioned against the pitfalls of excessive safeguards. "The more guardrails and artificial business rules you put into the system, the worse off," he noted. Such rules can be brittle and can skew user feedback, leading to incorrect learning. While guardrails are a necessity, the goal is to minimize their impact over time.</p> <p> Expedia Group's approach to governing AI involves three layers: principles first, followed by processes and tools, and finally automation. Principles guide decision-making across the organization. Processes and tools provide the necessary enforcement for these principles, with automation supporting the overall framework. Checkpoints, calibrated by risk level, such as evaluation rounds and security reviews, ensure governance is applied appropriately.</p> <h2>System Design and Specialized Agents</h2> <p> Amatriain advocated for the use of specialized agents over a monolithic AI. "I think it's much better to think of it as composition, having specialized agents that are very good at some task and then composing the system out of those specialized agents." This approach also enhances security, as individual agents can be evaluated and secured in isolation before being combined. Expedia emphasizes that the user must retain the final decision-making authority for actions like booking hotels—a critical security measure.</p>