Integrating design perspectives into AI evaluation processes
Arin Bhowmick, Chief Design Officer at SAP, argues that AI evaluation should move beyond technical metrics to include design-driven insights to better serve end users.
Current AI evaluation processes are largely driven by engineers and product managers who rely on scripts and performance dashboards. Arin Bhowmick, Chief Design Officer at SAP, contends that this approach creates a disconnect between technical functionality and actual user needs, potentially leading to products that perform well in tests but fail in real-world application.
Bhowmick outlines that AI evaluations fundamentally ask what constitutes a 'good' experience for the user. While data metrics are essential, he argues that designers possess the necessary judgment to identify nuances that software testing might overlook, such as conversational tone, jargon, and how an AI should respond when it lacks a definitive answer.
Furthermore, the article notes that many 'eval failures' are actually design problems that cannot be solved simply by tweaking a prompt or switching models. By incorporating usability testing and user research, designers can identify whether a failure stems from a technical glitch or a flawed user interaction model.
Ultimately, Bhowmick suggests that design leaders must be involved in setting evaluation criteria early in the development cycle. Defining user-centric boundaries for AI behavior is presented as an essential step in ensuring technology remains both useful and trustworthy.