The rules you write for yourself
India is making AI verification a duty for banks and courts. But the rest of us have to make the choice for ourselves.
In December of 1984, over 500,000 people in the city of Bhopal in Madhya Pradesh, India, were exposed to the highly toxic gas methyl isocyanate, in what is considered the world’s worst industrial disaster (Bhopal disaster). Estimates vary on the death toll but government sources put it at approximately 8,000 deaths and 558,125 injuries.
Earlier this month, the Supreme Court of India compared AI-hallucinated material to the release of methyl isocyanate into the province of law and justice: “invisible, insidious, and catastrophic by the time anyone notices.”
What prompted this?
The Case has prompted the Supreme Court of India to urge all Courts to adopt a zero-tolerance mode for producing, citing or using AI-generated precedents without verification.
It wasn’t the counsel who relied on AI-generated material. It was the tribunal adjudicating the case, whose own judgement rested on precedents that did not exist, with some fabricated paragraphs wrongly attributed to genuine citations.
The verification failure belonged to the adjudicator, not the advocate. But the knock on effect is significant as noted by the Supreme Court Judges who stated, “Judicial process and the judgment under challenge are tainted by the usage of materials which are said to be precedents, but in reality, they are unreal, fake, and do not exist at all. A decision of a Court or an adjudicating authority based on material which is fake and hallucinated is no decision at all, and it amounts to subversion of the rule of law. Such a decision is unsustainable and has to be set aside at the earliest.”
Just a few weeks before this case, the Reserve Bank of India (India’s Central Bank) drafted Guidance on Regulatory Principles for Model Risk Management, 2026. They named automation bias, over-reliance on model outputs, and decision fatigue as risks that human oversight must be designed against. The guidance also mandates kill switches, human-in-command arrangements, and non-delegable accountability. The draft is open now for comments until 24 July.
Within the span of a month, two separate institutions across two completely different contexts (Law and Finance) have arrived at the same conclusion: verification cannot be assumed, it has to be a designed gate in the workflow, and someone senior owns that it happened.
Both of these guidelines were driven by compliance and the risk of legal or financial harm. That is a good thing as far as it goes. People shouldn’t have their court case or their loan decided by hallucinated case history or a model that nobody checked. But neither the RBI nor the Supreme Court arrived here out of any concern for keeping the humans in the process capable. Skill retention is a side effect, not the reason for them.
And there is very little being drafted for those of us working in lower-stakes contexts. Most of us are not a bank or a court. I’ve not come across regulators mandating that a GTM team should ensure that a human has to be the one closing the deal.
Which means the preservation of our skills, and the sustainable use of AI, has to come from somewhere else. From the leaders and individuals designing our own workflows, deciding for ourselves where a human stays in the loop.
The RBI has effectively written down a standard: anyone reviewing an AI output should be able to challenge it, override it, or escalate it. But if that is the bar for a financial model, why is it any lower for the forecast your sales team generates, or the messaging your marketers no longer draft themselves? That is on us to decide.