Author : Tanisha
College: Bharat College Of Law, Kurukshetra University
To the Point
Artificial Intelligence (AI) is rapidly becoming part of the legal profession. Lawyers, researchers and courts can use AI-based tools to search information, summarise documents and assist with legal research. However, increasing dependence on AI has also created a serious concern: what happens when AI produces a case law or legal proposition that does not actually exist?
This problem is commonly known as an AI hallucination. In simple terms, an AI system may generate information that appears legally convincing but is factually incorrect, including imaginary judgments, incorrect citations or paragraphs that were never written by a court.
The issue is no longer merely theoretical. On 2 July 2026, the Supreme Court of India dealt directly with this problem in Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd., where the Court found that the NCLT had relied upon non-existent and AI-generated case law. The Supreme Court set aside the decisions of the NCLT and NCLAT and emphasised that AI may assist adjudication, but human beings must retain control over the judicial process.
Use of Legal Jargon
Judicial integrity refers to the reliability, independence and authenticity of the judicial process. Precedent means a previous judicial decision that may guide a court in deciding a similar legal issue. AI hallucination refers to an AI-generated response containing false, fabricated or non-existent information. Human oversight means that AI-generated material must be examined and verified by a qualified human before it is relied upon in legal proceedings.
Therefore, AI should be treated as a legal research aid and not as an independent source of law.
The Proof
The most important evidence of this emerging problem is the Supreme Court’s judgment in Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd., 2026 INSC 668. The dispute originated from insolvency proceedings involving Essel Infraprojects Ltd. The NCLT had relied upon six judicial citations while deciding the matter. When the issue reached the Supreme Court, it was pointed out that several of those citations were fake or contained paragraphs that could not be traced to the actual judgments.
The Supreme Court independently examined the cited authorities. It found that some citations were completely non-existent, while others contained genuine case names or citations but attributed non-existent paragraphs to those judgments. The Court also noted that the allegedly fake judgments had escaped scrutiny even at the appellate stage.
This demonstrates the central danger of AI-generated legal research: a citation may look perfectly genuine while actually being fabricated. For a legal system based upon precedents and reasoned judgments, such an error is not a minor technical mistake. It can directly affect the administration of justice.
Abstract
Artificial Intelligence has the potential to transform the legal profession by making legal research faster and more accessible. However, its increasing use also creates new challenges for judicial integrity. AI systems can generate incorrect legal propositions, fabricated authorities and non-existent case citations with considerable confidence.
The Supreme Court of India’s decision in Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd. provides an important warning against the unverified use of AI-generated legal material. The Court adopted a zero-tolerance approach towards fake or hallucinated precedents and clarified that legitimate use of AI is not prohibited. Rather, the concern is the presentation or reliance upon fabricated material as genuine legal authority.
This article examines the advantages and risks of AI in legal research, the implications of AI hallucinations for judicial decision-making and the need for responsible human supervision.
Case Law
Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd. (2026)
This is the central recent Indian authority on AI-generated legal hallucinations. The Supreme Court held that courts must adopt a zero-tolerance approach towards producing, citing or relying upon AI-generated precedents without verification. It stated that an advocate citing such material without verification may face professional consequences, while reliance upon such material by a judicial authority constitutes a serious lapse.
The Court further held that where fake or hallucinated material enters the decision-making process, the resulting decision cannot be sustained merely because the fake material did not directly determine the final outcome.
Importantly, the Court did not reject AI itself. It recognised that AI can assist the legal system but emphasised that there must be a human being in control at every stage of adjudication. The Court also directed the Bar Council of India to constitute a committee to consider appropriate guiding principles and disciplinary consequences regarding the submission of AI-generated fake precedents before courts.
AI: Threat or Tool for the Legal Profession?
AI itself is not necessarily a threat to the legal system. In fact, it can be extremely useful. It can assist lawyers in locating authorities, identifying relevant legal provisions, organising large documents and summarising lengthy material. For law students and young advocates, it can also make preliminary research considerably faster.
The problem begins when convenience turns into blind dependence. A lawyer who simply copies an AI-generated citation without checking the original judgment risks misleading the court. Similarly, a judge or tribunal cannot outsource the responsibility of judicial reasoning to an AI system.
The Supreme Court recognised this distinction. AI can assist human effort, but it becomes dangerous when it starts functioning as an alternative to human thinking and decision-making.
Need for Human Verification
The safest approach is not to completely avoid AI but to establish a system of mandatory human verification. Before relying upon AI-generated legal research, a lawyer or researcher should verify whether the case actually exists, whether the citation is correct, whether the quoted paragraph appears in the original judgment, whether the judgment remains good law, whether it has been subsequently overruled or distinguished, and whether its factual context actually supports the proposition being cited.
Legal research is not simply about finding words that match a particular issue. Law requires context, interpretation and reasoning.
The Way Forward
India does not need to choose between technological development and judicial integrity. The better approach is responsible integration. Clear professional guidelines should be developed for the use of generative AI by advocates and judicial officers.
Lawyers should remain professionally accountable for every authority submitted before a court, irrespective of whether it was found through traditional research or AI assistance. Courts and tribunals should also develop appropriate technological safeguards for identifying suspicious or unverifiable citations.
Finally, legal education must adapt. Law students should be taught not merely how to use AI but also how to question and verify AI-generated information. The Supreme Court’s direction to the Bar Council of India is significant because it moves the discussion towards professional accountability and regulation.
Conclusion
Artificial Intelligence is likely to become an important part of India’s legal system. Its ability to process large amounts of information can improve efficiency and reduce the burden of routine legal research. However, speed cannot come at the cost of accuracy.
The judgment in Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd. sends a clear message: AI may assist the administration of justice, but it cannot replace human responsibility for the truthfulness of legal material.
The real challenge is therefore not whether courts should use AI, but how they should use it without surrendering human judgment. A legal system ultimately depends upon authentic laws, genuine precedents and reasoned decisions. AI can help lawyers and judges reach these sources faster, but it must never be allowed to decide what is true merely because its answer sounds convincing.
The future of AI in Indian courts should therefore follow one simple principle: technology may assist justice, but responsibility must remain human.
FAQ
1. Can lawyers use AI for legal research?
Yes. AI can be used as a research and drafting aid, but the information generated by AI must be independently verified before being relied upon in legal proceedings.
2. What is an AI hallucination in law?
An AI hallucination occurs when an AI system generates false or fabricated information, such as a non-existent judgment, incorrect citation or imaginary paragraph.
3. What did the Supreme Court hold in Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd.?
The Supreme Court adopted a zero-tolerance approach towards unverified AI-generated fake or hallucinated precedents and set aside the decisions affected by such material.
4. Has the Supreme Court banned the use of AI in courts?
No. The Court did not ban legitimate use of AI. Its concern was the presentation or reliance upon fake or hallucinated material as genuine precedent.
5. Who is responsible for verifying AI-generated legal research?
The lawyer, researcher or judicial officer relying upon the material remains responsible for verifying its authenticity. AI cannot replace professional and judicial responsibility.
6. What is the future of AI in Indian courts?
AI is likely to remain an important technological aid in legal research and court administration. Its responsible integration will require human supervision, professional accountability and appropriate regulation.
