AI Agent Evaluation Shifts From Single Conversations to Cohort Analysis

The evaluation of AI agents is undergoing a significant change, as a seemingly flawless single conversation can still mask underlying product issues. This gap is prompting enterprises to move away from scoring individual interactions toward comparing cohorts of users against a baseline.

At VB Transform 2026, industry leaders Harrison Chase (LangChain CEO), Hui Zhang (Conviva CTO), and Emmanuel Turlay (CoreWeave engineering director) discussed this transition. The industry is moving toward cheaper, narrower judge models, shifting the focus from isolated trace scoring to broader comparative analysis.

"We sometimes see teams that have almost eval paralysis," said Harrison Chase. He described evaluation criteria as a living specification, akin to a product requirements document (PRD), defining what an agent should and should not do. This approach contrasts with static, pre-launch test suites that can miss critical issues.

Emmanuel Turlay echoed this sentiment, noting that aiming for 100% test coverage did not prevent production bugs. "Broad, always-on monitoring catches more real failures than an exhaustive pre-launch test suite," Turlay stated. The recommended approach involves setting up wide online checks first, identifying failure classes, and then building targeted offline evaluations.

Hui Zhang emphasized the importance of "contrastive analysis," which compares groups of users against a baseline rather than scoring individual traces. An example involving shoe purchases illustrated how isolated successes can hide issues like increased clarification requests or off-conversation purchases, only visible through larger-scale data analysis. The industry faces the challenge of balancing scalable automated evaluation with necessary human oversight, especially in sensitive sectors like finance and healthcare.