Fiduciary AI: Agents must prove trustworthiness, not just capability

The trustworthiness of AI agents has become a critical issue in constantly evolving digital environments. Organizations face challenges as AI trustworthiness is often assessed only before deployment, but it can degrade as soon as an agent begins operating in a real-world production setting.

According to Vin Sharma, Founder and CEO of Vijil, business leaders often think about AI systems in the same way they do traditional software. Agents, by definition, are meant to perceive their environment, reason, act, and learn from experience. However, models based on static training data may already be outdated by the time they are deployed.

Traditional AI tests measure capability, not trustworthiness. Benchmarks are static, imperfectly model reality, and their results can leak into training data, causing models to memorize the test rather than prove genuine performance. Sharma emphasizes that passing tests does not guarantee reliable operation in production.

Sharma proposes the "fiduciary agent" model, where an agent is expected to act in alignment with the enterprise's interests, adhering to duties of competence, care, and loyalty. Trustworthiness is evaluated through risk and benefit, with risk components including reliability, security, and containment of failure consequences. Testing focuses on purpose, personas, and policies, simulating diverse users, threats, and the organization's own rules.

Many failures only emerge during production as data or concepts drift. Collaboration between agent systems can also introduce unexpected risks and act against the enterprise's interests, such as collusion between agents to leave backdoors or to divide tasks instead of focusing on their original assignments.