Every privacy professional learns purpose limitation early. Data collected for one reason should not quietly become fuel for another. It is one of the oldest ideas in the field, and the arrival of foundation models has put it under more pressure than any other principle.
The pressure is easy to understand. An organization holds years of customer data, and a model trained on it could be valuable. The data feels like an asset already paid for. It is not. It was collected under a stated purpose and a set of expectations, and training on it is a new use that has to be justified on its own terms.
In the United States, California’s regulations say it most plainly. Under 11 CCR §7002, processing must be reasonably necessary and proportionate to the purpose for which the data was collected, or to another disclosed purpose that is compatible with the context of collection. Compatibility turns on the consumer’s reasonable expectations at the time of collection and “the strength of the link” between the original purpose and the new one.
The regulation even gives an example that reads like a training case. It describes a weak link between data collected to provide a cloud storage service and “the use of the information to research and develop an unrelated facial recognition service.” Where the link fails, the business needs consent.
The Federal Trade Commission’s staff made a related point in February 2024. They warned that it may be unfair or deceptive for a company to adopt more permissive data practices, such as using data for AI training, and to tell consumers only through “a surreptitious, retroactive amendment” to its terms. In their words, “A business that collects user data based on one set of privacy commitments cannot then unilaterally renege.” The FTC’s broader posture on AI has shifted since then, but that point rests on deception law far older than AI, and the post is still published.
Europe runs the same test with a longer list of factors. Article 6(4) of the GDPR weighs the link between purposes, the context of collection, the nature of the data, the possible consequences for people, and the safeguards in place.
Two episodes show what the compatibility test looks like in practice.
In 2023, a change to Zoom’s terms of service, made months earlier, was read by users as permitting AI training on their calls. The backlash forced a public correction, and the company’s chief executive called it a process failure. Zoom now states that it does not use customer audio, video, chat or other communications to train AI models.
In 2024, Meta paused plans to train models on content from European users after engagement with the Irish Data Protection Commission. When training resumed in May 2025, the commission’s statement listed the changes Meta had made: clearer notices, an easier way to object, de-identification and filtering, and an updated compatibility assessment.
The lesson is not that training is forbidden. It is that the compatibility assessment ends up being the instrument either way. It can be done before training, on the organization’s own terms, or after a public correction, on someone else’s.
Most privacy programs have no moment where this question is forced. Data enters a training pipeline the way it enters an analytics warehouse, through an engineering ticket.
The fix is a recorded decision that sits before any training run on personal data. It should name:
- the data and the purpose stated when it was collected, in the words used at the time;
- the new purpose, specifically;
- the reasoning on compatibility, including the strength of the link and what a reasonable person in that relationship would expect;
- the safeguards that narrow the gap, such as de-identification, aggregation, filtering or an opt-out;
- who decided, and when.
In California this is no longer optional for some uses. The risk assessment rules that took effect on January 1, 2026 require an assessment before processing personal information to train automated decisionmaking technology used for significant decisions, or to train facial recognition, emotion recognition or identity verification technology.
I once gave an AI agent a read-only audit with a single permitted output: one report file. Partway through, its own tooling told it to save an extra file to manage its working memory. The agent declined, because its assignment allowed one file, and stopped with an honest partial report rather than a complete one that broke scope.
That is purpose limitation working where it now has to work: inside the system, at the moment of the decision, not only in a policy document.
The decision belongs at data sourcing, before training. After training, the data is in the model, and the options narrow to retraining or arguing that the model holds no personal data. The next essays in this series cover both of those problems. Neither is cheap.
- 1.What purpose did we state when we collected this data, in the words we used then?
- 2.Would a reasonable person in that relationship expect this use?
- 3.Is any of the data sensitive, and does this use need consent?
- 4.What safeguards narrow the gap between the old purpose and the new one?
- 5.Who signed the decision, and where is it filed?
The question is rarely whether training is allowed. It is whether anyone asked before the model was trained.