High-Reliability Organizations Offer Lessons for the AI Era
Leaders must build trust and accountability into AI processes from the start. System failures can occur if AI outputs are mishandled or misunderstood.
A radiology conference incident highlighted a critical flaw in AI implementation: a system failure not of the technology itself, but of the human processes surrounding it. As reported by Inc. Magazine, an AI tool designed to triage imaging studies at a hospital was mistakenly made visible to emergency department physicians. These physicians began relying on the AI's preliminary read, leading to significant delays for other patients, as the tool's output was not yet validated by a radiologist.
Angela Adams, CEO of Inc. Magazine and formerly a critical care nurse, points out that the AI performed its programmed function. The failure lay in the absence of clear protocols defining who could trust the AI's output, when, and with what oversight. This scenario exemplifies what Adams identifies as the primary risk of the AI era: organizations adopting AI faster than they establish the necessary trust and accountability structures for its safe use.
Adams draws parallels to High-Reliability Organizations (HROs) studied in high-risk industries like aviation and nuclear power. These organizations prioritize preventing catastrophic failures by rigorously hunting for near-misses and treating anomalies as early warnings. This focus on reliability, Adams argues, paradoxically allows for faster, more decisive action under pressure.
In contrast, many companies approach AI by treating safety and governance as an afterthought. The article cites Klarna's experience, where an AI assistant initially praised for efficiency later faced criticism for a decline in customer service quality, necessitating the reintroduction of human staff. This illustrates that speed and cost savings alone do not capture the full value or potential risks of AI deployment.
The core message from Adams is that true trust in AI outcomes resides with the individuals who must act upon them. By adopting principles from HROs—anticipating failure, deferring to those closest to the work, and building robust oversight—organizations can mitigate AI-related risks and ensure responsible technological advancement.