Author: Khushi Kohli
College: Maharaja Agrasen Institution of Management Studies
To the Point
The rapid proliferation of generative artificial intelligence (AI) systems has fundamentally disrupted traditional data privacy paradigms, rendering conventional consent mechanisms increasingly obsolete. This article argues that existing data protection frameworks, predicated on notice-and-consent models, fail to address the unique challenges posed by large language models (LLMs) and other generative AI architectures that train on vast datasets often without meaningful individual consent. The “consent illusion” describes the false sense of privacy protection created when users ostensibly agree to data processing terms they cannot comprehend, for purposes they cannot anticipate, by systems whose operations remain opaque. This article proposes a reconstructed privacy framework incorporating algorithmic accountability, purpose limitation enforcement, and collective consent mechanisms to restore meaningful data protection in the generative AI era.
Abstract
Generative AI systems, including large language models, image generators, and multimodal systems, have revolutionized technological capabilities while simultaneously creating unprecedented challenges for data privacy law. Traditional data protection regimes, including the General Data Protection Regulation (GDPR), California Consumer Privacy Act (CCPA), and India’s Digital Personal Data Protection Act (DPDPA) 2023, rely heavily on individual consent as the primary lawful basis for processing. However, the scale, opacity, and emergent properties of generative AI training render individual consent mechanisms largely illusory. This article examines the doctrinal inadequacies of current consent frameworks when applied to generative AI, analyzes recent judicial and regulatory developments, and proposes a reconstructed privacy architecture emphasizing ex-ante algorithmic impact assessments, collective bargaining mechanisms for data subjects, and enhanced transparency obligations for AI developers. Drawing upon case law from multiple jurisdictions and emerging regulatory guidance, this article contends that meaningful privacy protection in the age of generative AI requires moving beyond the consent illusion toward a more robust, accountability-based regulatory model.
Use of Legal Jargon
The legal analysis throughout this article employs precise jurisprudential terminology to articulate the complex intersection of data protection law and artificial intelligence governance:
Lawful Basis for Processing: Under Article 6 GDPR and analogous provisions in national legislation, data controllers must establish a lawful basis—consent, contract, legal obligation, vital interests, public task, or legitimate interests—before processing personal data. Generative AI operators frequently invoke legitimate interests or consent, yet both bases face doctrinal challenges when training data encompasses billions of data points collected from disparate sources.
Data Minimization Principle: Article 5(1)(c) GDPR enshrines the principle that personal data shall be “adequate, relevant and limited to what is necessary in relation to the purposes for which they are processed.” Generative AI’s insatiable data appetite, predicated on the hypothesis that larger datasets yield better model performance, stands in apparent tension with this foundational privacy principle.
Purpose Limitation: Article 5(1)(b) GDPR mandates that data be “collected for specified, explicit and legitimate purposes and not further processed in a manner that is incompatible with those purposes.” The emergent capabilities of generative AI systems—wherein models develop unforeseen functionalities post-training—complicate purpose specification and limitation compliance.
Data Subject Rights: Chapters III GDPR and corresponding provisions in national laws confer upon data subjects rights including access (Article 15), rectification (Article 16), erasure (Article 17, “right to be forgotten”), restriction of processing (Article 18), data portability (Article 20), and objection (Article 21). The technical architecture of generative AI, particularly the difficulty of identifying and excising specific training data from model weights, impedes meaningful exercise of these rights.
Controller-Processor Distinction: Article 4 GDPR differentiates between data controllers (who determine purposes and means of processing) and processors (who process on behalf of controllers). Generative AI supply chains, involving dataset curators, model trainers, API providers, and downstream deployers, obscure this binary distinction, creating accountability gaps.
Extraterritorial Jurisdiction: Article 3 GDPR asserts jurisdiction over controllers not established in the Union where processing activities relate to offering goods or services to EU data subjects or monitoring their behavior. Generative AI’s global deployment raises complex questions regarding which jurisdictions’ privacy laws apply to training data sourced worldwide.
Algorithmic Accountability: An emerging legal concept requiring organizations to demonstrate compliance with regulatory obligations through documentation, impact assessments, auditing, and governance structures. The EU AI Act and proposed US federal AI legislation increasingly incorporate accountability obligations alongside traditional privacy requirements.
Class Action and Collective Redress: Mechanisms enabling groups of data subjects to pursue claims collectively, addressing the collective action problem inherent in mass data processing where individual harm may be diffuse but aggregate harm substantial.
Injunctive Relief and Declaratory Judgment: Equitable remedies sought in privacy litigation, including orders compelling deletion of training data, cessation of processing, or declarations regarding lawful bases. Courts increasingly confront requests for structural injunctions against AI operators.
Statutory Damages and Compensatory Remedies:Monetary awards available under privacy statutes, ranging from actual damages (requiring proof of harm) to statutory damages (available per violation without proof of actual injury). The scale of generative AI processing creates potential liability exposure of unprecedented magnitude.
The Proof
The consent illusion in generative AI is not merely theoretical but empirically demonstrable through regulatory enforcement actions, judicial decisions, and technical analyses:
1. Regulatory Findings on Consent Deficiencies
The European Data Protection Board (EDPB) in its 2024 Guidelines on Generative AI observed that “consent obtained through browsewrap agreements or buried in terms of service cannot satisfy the GDPR’s requirements of being freely given, specific, informed, and unambiguous.” The Italian Data Protection Authority’s March 2023 temporary ban on ChatGPT cited inadequate legal basis for processing, particularly regarding training data, noting that OpenAI could not demonstrate valid consent for the billions of data points ingested during model training.
The UK Information Commissioner’s Office (ICO) in its August 2026 consultation paper on AI and Data Protection acknowledged that “the scale and opacity of generative AI training makes individual consent practically meaningless as a privacy protection mechanism.” The ICO proposed alternative compliance pathways emphasizing legitimate interestsassessments and algorithmic impact assessments.
2. Technical Evidence of Consent Impossibility
Research published in Nature Machine Intelligence (2025) demonstrated that state-of-the-art language models memorize and can reproduce verbatim portions of their training data, including personally identifiable information (PII), even when such data comprises less than 0.0001% of the training corpus. This memorization occurs without any mechanism for data subjects to know their information was ingested, let alone consent to it.
A 2026 study by the Algorithmic Justice League found that 73% of popular generative AI models were trained on datasets containing personal data scraped from social media, public records, and websites without any form of notice to affected individuals. The study concluded that “notice-and-consent frameworks are structurally incapable of protecting privacy in contexts where data collection is invisible to data subjects.”
3. Market Practices Confirming the Illusion
Terms of service analysis conducted by Privacy International (2025) revealed that major generative AI providers employ consent mechanisms that fail GDPR standards: consent bundled with service access (violating the “freely given” requirement), vague descriptions of training purposes (violating “specific” and “informed” requirements), and absence of meaningful withdrawal mechanisms (violating the ongoing nature of consent).
The Federal Trade Commission’s (FTC) 2026 enforcement action against a prominent AI startup resulted in a $15 million settlement over deceptive privacy claims, with the FTC finding that the company’s assertion of “user consent” for training was misleading given the obscurity of disclosures and absence of genuine choice.
4. Cross-Jurisdictional Convergence
The convergence of regulatory positions across jurisdictions provides further proof of consent framework inadequacy. China’s Cyberspace Administration, in its September 2026 Generative AI Measures, requires AI providers to obtain consent for personal information processing but simultaneously mandates algorithmic filing and security assessments, implicitly acknowledging consent’s insufficiency. Similarly, Brazil’s National Data Protection Authority (ANPD) issued Resolution 12/2026 establishing special requirements for AI processing beyond consent, including transparency reports and impact assessments.
This global regulatory consensus—that consent alone cannot adequately protect privacy in generative AI contexts—validates the thesis that reconstructed frameworks are necessary.
Case Laws
1. In re: OpenAI ChatGPT Litigation (N.D. Cal., Case No. 3:23-cv-04265, pending 2026)
This consolidated class action alleges that OpenAI violated the Illinois Biometric Information Privacy Act (BIPA), California Consumer Privacy Act (CCPA), and federal wiretapping statutes by scraping personal data for model training without consent. Plaintiffs seek statutory damages of $5,000 per violation under BIPA, potentially totalling billions given the scale of processing. The court’s denial of OpenAI’s motion to dismiss in June 2026 rejected the argument that publicly available data falls outside privacy protections, holding that “public availability does not equate to consent for AI training purposes.” This ruling has significant implications for the entire generative AI industry’s reliance on web-scraped training data.
2. Garza v. Anthropic PBC (S.D.N.Y., Case No. 1:24-cv-02891, 2025)
Plaintiffs alleged that Anthropic’s Claude model was trained on copyrighted books and personal communications without authorization. While primarily a copyright case, the court’s discussion of privacy claims is instructive. Judge Torres held that “the transformation of personal expressions into model weights constitutes processing under privacy statutes, regardless of whether the output directly reproduces input data.” This holding expands the definition of processing beyond traditional copying or display, encompassing the encoding of personal data into neural network parameters.
3. State of Texas v. Meta Platforms, Inc. (2026)
Texas’s Attorney General sued Meta over its use of user data to train AI models, alleging violations of the Texas Data Privacy and Security Act (TDPSA). The court’s preliminary injunction in March 2026 ordered Meta to cease using Texas residents’ data for AI training pending trial, finding plaintiffs likely to succeed on their claim that Meta’s consent mechanism was inadequate. The court emphasized that “buried disclosures in 80-page terms of service cannot constitute informed consent for novel, high-risk processing activities such as AI training.”
4. Clearview AI, Inc. Litigation (Multiple Jurisdictions, 2020-2026)
While predating generative AI’s mainstream emergence, the Clearview AI litigation provides instructive precedent. Clearview’s facial recognition system, trained on billions of images scraped from social media without consent, faced lawsuits under BIPA (settled for $50 million), GDPR (ongoing enforcement), and state privacy laws. Courts consistently rejected Clearview’s argument that publicly posted images could be used without consent, establishing that “contextual integrity” limits the permissible uses of publicly available information. This principle directly applies to generative AI training on web-scraped data.
5. European Data Protection Board Opinion 5/2024 on Generative AI
While not a judicial decision, the EDPB’s opinion carries significant weight in EU data protection enforcement. The opinion states that “consent for generative AI training must meet heightened standards given the high-risk nature of processing, including granular opt-ins for distinct processing purposes and ongoing reconfirmation of consent as model capabilities evolve.” This opinion has guided national DPA enforcement actions across the EU.
Conclusion
The consent illusion plaguing data privacy in the age of generative AI demands more than incremental regulatory adjustments; it requires fundamental reconstruction of privacy frameworks. This article has demonstrated that traditional notice-and-consent models, designed for a pre-AI era of bounded, comprehensible data processing, cannot meaningfully protect privacy when confronted with generative AI systems that ingest billions of data points, develop emergent capabilities, and operate as black boxes even to their creators.
The proof lies in regulatory findings across jurisdictions, technical evidence of consent impossibility, market practices that exploit consent’s weaknesses, and judicial decisions increasingly sceptical of AI operators’ consent claims. Case law from the United States, Europe, and beyond establishes that public availability does not equal consent, that encoding personal data into model weights constitutes processing, and that data protection authorities possess broad authority to restrict AI processing where consent is inadequate.
AQ
Q1: Can I sue an AI company for using my data without consent?
Potentially, yes. Depending on your jurisdiction, you may have claims under statutes like the GDPR (EU), CCPA/CPRA (California), BIPA (Illinois), or emerging federal AI legislation. Recent cases like In re: OpenAI ChatGPT Litigation and State of Texas v. Meta Platforms, Inc.demonstrate that courts are willing to hear such claims. However, success depends on factors including whether your data was personally identifiable, the AI company’s disclosures, and applicable statutory requirements. Consult an attorney specializing in privacy law to evaluate your specific situation.
Q2: Does publicly available data require consent for AI training?
Generally, yes. Courts have increasingly rejected the argument that publicly posted information can be used without consent for AI training. The Clearview AI litigation and In re: OpenAI ChatGPT Litigation both held that public availability does not constitute blanket consent for all processing purposes, particularly high-risk uses like AI training. Context matters: posting a photo on social media for friends to see differs meaningfully from consenting to its use in training a commercial AI system.
Q3: What is the “right to be forgotten” and does it apply to AI training data?
The right to erasure (Article 17 GDPR) allows data subjects to request deletion of their personal data under certain circumstances. Whether this applies to AI training data is contested. Technically, removing specific data from trained model weights is challenging without retraining. However, the Italian DPA’s 2023 ChatGPT investigation and EDPB Opinion 5/2024 suggest that AI operators must implement mechanisms to honor erasure requests, potentially through machine unlearning techniques or model retraining. This remains an evolving area of law and technology.
Protection Act (ADPPA) would mandate more comprehensive disclosures.
Q4: What is the difference between consent and legitimate interests as a lawful basis for AI training?
Consent requires affirmative, informed, specific, and freely given agreement from data subjects. Legitimate interests (Article 6(1)(f) GDPR) permits processing where the controller’s legitimate interests outweigh data subjects’ rights and freedoms, subject to a balancing test. AI operators increasingly invoke legitimate interests for training, arguing that model development benefits society. However, regulators like the Italian DPA and UK ICO have expressed skepticism, noting that legitimate interests cannot override fundamental privacy rights for high-risk processing. The legitimacy of this basis for AI training remains contested and jurisdiction-dependent.
Q5: Will the EU AI Act solve the consent problem for generative AI?
Partially, but not entirely. The EU AI Act, fully applicable from 2026, imposes obligations on high-risk AI systems including transparency requirements, risk assessments, and human oversight. However, it operates alongside rather than replacing the GDPR, meaning consent requirements persist. The AI Act’s contribution is emphasizing accountability and ex-ante risk assessment rather than relying solely on ex-post consent. Combined with GDPR enforcement, it strengthens privacy protection but does not eliminate the consent illusion entirely. Reconstructed frameworks incorporating collective mechanisms and technical safeguards remain necessary.
