Zillow's Engineering Chief: AI ROI Requires Measurement Before Building

Zillow's Senior Vice President of Engineering, Toby Roberts, stated that the return on investment for artificial intelligence initiatives can only be validated if measurements are in place before development commences. At the VB Transform 2026 event, Roberts highlighted Zillow's approach, emphasizing that the difficulty lies not in data acquisition but in creating persistent context throughout a customer's complete journey.

Roberts explained that Zillow's real estate clients interact through various channels and professionals, such as a phone screen, loan officer, and real estate agent, often over extended periods. They expect continuity in information. A single chatbot cannot manage this complexity, prompting Zillow to develop its own AI architecture capable of carrying context across this entire process.

Initially, Zillow's AI efforts focused on establishing a robust data foundation, incorporating a data mesh approach, clear data lineage, and a governance structure with permissions. However, the team found that building a system to remember and maintain customer journey context across different touchpoints was the more significant challenge.

Instead of routing customers through a single model API, Zillow built its own harness. This approach leverages the company's two decades of machine learning experience, utilizing smaller, task-specific fine-tuned models over a single general-purpose one. Zillow now employs thousands of Glean agents internally for repetitive tasks, integrated via the Glean MCP gateway to centralize efforts.

Roberts stressed that attributing significant gains, like Zillow's reported 40% increase in shipped code attributed to AI, relies on pre-existing measurement frameworks, such as DORA metrics. He advised enterprises to centralize context management to avoid duplicated effort and to treat context as a cost lever for efficiency.