In a recent interview, Steve Hasker, chief executive of Thomson Reuters, said, “Fiduciary professions like law, tax, audit and compliance require more than…accelerating everyday knowledge work. What ultimately matters is whether the output is authoritative, traceable and accountable to professional standards. In high-stakes environments, speed alone isn’t the differentiator. Trust is.”
As artificial intelligence (AI) becomes an increasingly disruptive force in white‑collar work, trust has emerged as the central point of tension in highly regulated industries such as insurance, where accuracy and accountability are paramount. That trust is built through transparency, and transparency depends on explainability. Explainability is therefore no longer a technical nuance; it has become a business necessity. It now sits at the core of governance, regulatory compliance, litigation readiness, and customer trust. The EU AI Act, for example, mandates robust transparency, explainability, and human oversight for AI systems deemed high risk in insurance. By contrast, the United States relies on a fragmented, state‑led regulatory approach anchored by the NAIC Model Bulletin, emphasizing fairness and accountability in claims and underwriting. As of April 1, 2026, 25 U.S. states have implemented NAIC Model Bulletin of the use of AI systems by insurers.
Importantly, while AI is increasingly deployed in underwriting, pricing, claims, and fraud detection, it does not alter insurers’ underlying legal responsibilities. State insurance laws remain fully applicable whether decisions are made by humans, automated systems, or external vendors.
Across the property and casualty landscape, the appeal of AI is easy to understand as carriers want to improve consistency, lower handling costs, and help claims professionals manage rising complexity. The real divide, however, is not between using AI or not, but between models that produce mere answers and those that can explain them.
The question carriers should be asking is not whether AI can improve speed or efficiency, but whether the decisions AI helps produce can be explained to customers and regulators with confidence. For instance, when a claim is denied, delayed, or flagged for possible fraud, insurers would need to show not only what decision was made, but why. For claims leaders, CIOs, and fraud and risk executives, that is where the real cost of black box AI models begins to come into focus.
Black box models built on XGBoost and other neural networks can be appealing for their speed and strong accuracy, but their lack of transparent reasoning creates risks that are easy to miss initially and far harder to manage once they emerge in audits, escalations, regulatory scrutiny, or litigation. For example, we witnessed this exact issue at a mid-size carrier that deployed a neural fraud model and faced a regulator inquiry 18 months later resulting in a fine because the AI model relied on data with systemic bias.
Where the Friction Starts
Claims decisions can have long lives. They are reviewed, questioned, reopened, and sometimes taken to court often months or years after the initial adjudication. Audit and compliance teams must demonstrate consistency across similar claims, because regulators expect clear accountability for decisions that affect policyholders financially.
However, if the underlying AI system cannot explain how it reached a conclusion, each of those reviews becomes harder than it needs to be. As a result, claims teams are forced to reverse‑engineer decisions from technical outputs, resulting in explanations that are, at best, an approximation of the real decision logic. This puts insurers at a disadvantage: it slows decision review, intensifies scrutiny, and creates avoidable exposure with regulators and opposing counsel. Automated claims decisions should be defensible on their face; not rebuilt through forensic analysis after the fact.
Why Explainability Cannot Be Retrofitted
A common belief among carriers evaluating AI models is that explainability can be addressed later—bolted on once the model is performing and the business case is proven. In practice, that assumption is far more costly than it appears.
Once a claims process is built around neural network models that cannot explain their own reasoning, meaningful transparency becomes expensive to introduce and almost always remains incomplete. Post-hoc techniques such as SHAP values or LLM-generated explanations can describe what an output looks like, but they do not reconstruct the actual basis for the decision. For a claims adjuster facing a regulatory inquiry or a disputed denial, that distinction is not academic; it is the difference between a defensible answer and an improvised one.
The problem compounds in dynamic claims environments. Repair costs shift, litigation patterns evolve, and fraud tactics adapt. When those changes interact with a black-box model, the effects on decision behavior are invisible until they have already caused harm. Subtle outcome drift can go undetected for months, surfacing only when inconsistency accumulates into a pattern of complaints, an uptick in escalations, or direct questions from a regulator about how similar claims are being treated.
A deployment we undertook for an auto claims fraud detection model in the Middle East illustrates this precisely. The legacy model the carrier had been using consistently flagged claims involving accidents after midnight on weekends as high-risk—a logic inherited from North American and European training data, where late-night weekend incidents correlate strongly with alcohol-impaired driving. In the country we were operating in, alcohol consumption is prohibited and families routinely stay out late into the evening on weekends. The model's risk signal was not just wrong; it was systematically wrong in a way that penalized a normal cultural behavior. Because the model could not explain its reasoning, no one inside the organization had caught it. The carrier only became aware of the problem when it began showing up as elevated regulatory scrutiny and measurable customer dissatisfaction.
That is the real danger of retrofitting explainability: by the time the problem is visible, it has already become a governance failure, a customer trust problem, and a regulatory exposure—all at once. Explainability is not a feature that can be scheduled for a later release. It needs to be present from the first decision the model influences.
The Case for Explainability by Design
The first question claims leaders should ask of any AI system influencing claim outcomes is not "how accurate is it?" but can my adjuster read the reasoning, and will it hold up if a regulator asks?" This is where interpretable AI differs fundamentally from black-box models retrofitted with explanation layers.
The regulatory environment is making this distinction unavoidable. The NAIC Model AI Bulletin, along with state-level mandates in New York, Colorado, and California, now requires insurers to demonstrate that AI-driven decisions are transparent, auditable, and free from proxy discrimination. New York's DFS Circular Letter 2024-7 goes further, requiring that vendors themselves be subject to audit and that explanatory documentation be maintained for adverse outcomes. For claims leaders, any AI system that cannot produce a traceable decision path on demand is already a liability in those jurisdictions.
What interpretable-by-design looks like in practice is worth being specific about, because "explainability" has become a marketing term that conceals significant variation.
The most basic test is whether the reasoning is produced as part of the decision itself, not generated afterward. When a fraud flag is raised or a claim is escalated, the system should surface the specific factors that drove that outcome—policy terms, documentation gaps, behavioral signals, claim timing anomalies—ranked by contribution, in language an adjuster can read without a data science intermediary. This is architecturally different from layering SHAP values or LLM-generated summaries on top of a neural network output. Those techniques approximate the decision; they do not represent it.
A related and equally important capability is whether the model's logic can be interrogated and overridden at the case level. An adjuster who disagrees with a recommendation should be able to see exactly which inputs drove it, assess whether those inputs are accurate and relevant to this specific claim, and document their override with confidence. Systems that allow override without explanation offer no improvement over gut instinct. Systems that show their reasoning make override a professional act of judgment, not a workaround.
Less visible but equally consequential is whether the system can detect and show its own drift. In a claims environment where fraud tactics, repair costs, and regional patterns shift continuously, a model that cannot flag changes in its own decision behavior creates the very governance gaps regulators are targeting. Interpretable models, particularly those built on structured decision logic rather than opaque neural architectures, make outcome drift visible before it accumulates into a compliance event.
None of this requires sacrificing predictive sophistication. Purpose-built claims AI models are already demonstrating that interpretable architectures can match or exceed the fraud detection accuracy of black-box alternatives, while producing the adjuster-readable, regulator-ready reasoning that carries a decision through its entire lifecycle from intake to litigation, if it comes to that.
This leaves claims leaders with a clear choice. The standard for explainability is no longer hypothetical; it is already here. Whether the AI models they are currently using meet that standard will ultimately be tested not by AI-led efficiency, but by regulators, courts, and customers. In claims, opacity is not merely a system limitation; it is a measurable business risk, and time only makes it more visible.
About the author:
Amer Kayani is co-founder and CEO, TAO Trees.