Deidentification Expectations in AI: A Refined Perspective

In October 2024, the Office of the Australian Information Commissioner (OAIC) released two significant guides addressing privacy in AI contexts: 

1. Guidance on privacy and the use of commercially available AI products (Guidance 1)
2. Guidance on privacy and developing and training generative AI models (Guidance 2)

These guides are timely coinciding with the OAIC’s preliminary enquiry into allegations that patient data albeit ‘deidentified’ from I-Med was used without consent to train Harrison.ai’s AI models. While the enquiry is in its early stages, the release of these guides signals the OAIC’s heightened focus on privacy compliance in AI practices. 

(We covered both of these Guides in our earlier blog post.)

This article explores the contentious issue of de-identification, particularly in the context of the development of AI models and as explored in the OAIC’s new guides. 

The controversy surrounding de-identification

De-identification raises critical questions:

• Can organisations de-identify your personal information without your consent? 
• What restrictions apply to de-identified information?
• What level of de-identification is sufficient to mitigate privacy risks? 

In Australia, de-identification serves as a catch-all term that varies by context. This contrasts with the EU GDPR, which distinguishes between: 

• Pseudonymous data: not fully de-identified and still regulated under privacy law.
• Anonymous data: requires an exceedingly high standard to ensure no reidentification is possible. 

In Australia, the Privacy Act 1988 does not define de-identification explicitly but embeds it within broader privacy protections. 

De-identification in the Australian Privacy Act

Key principles relating to de-identification in the Privacy Act include: 

1. Section 6 – definitions

• ‘Personal information’ refers to information or an opinion about an identified individual or an individual who is reasonably identifiable. 
• De-identified information falls outside this definition if information can no longer reasonably identify an individual 

2. Australian Privacy Principles (APPs)

• APP 4 (Dealing with unsolicited information): Requires entities to destroy or de-identify unsolicited personal information they cannot lawfully retain
• APP 6 (Secondary use): Enables compliant secondary uses of personal information (e.g., for public health or safety research) if de-identified 
• APP 7 (Direct marketing): De-identified information is generally exempt from restrictions on direct marketing, provided de-identification is robust. 
• APP 11 (Security of personal information): Encourages de-identification (over destruction) of personal information when it is no longer needed, especially when the de-identified information has potential future utility, such as for developing new products. 

Insights from the OAIC’s new AI guides

Guidance 1 – for users of AI

Though not primarily focused on de-identification, Guide 1 highlights compliance with APP 6 (Use or disclosure of personal information) when using personal information in AI systems. Key considerations include: 

• Ensuring the use of personal information aligns with its original collection purpose 
• Assessing whether training AI models on personal information or de-identified information, are within individuals’ reasonable expectations. 

• Seeking consent or offering a clear opt-out option when reasonable expectations cannot be established 

The guide also cautions:

1. De-identified or anonymised information in AI systems may still be at risk of reidentification 
2. Any personal information created using AI which your organisation is not permitted to collect under APP 3 (Collection of solicited personal information) must be destroyed or de-identified

Guidance 2 – for developers of AI

Guide 2 delves a little more deeply into deidentification, highlighting: 

• Effectiveness and risks 

• De-identification is not foolproof; steps to de-identify personal information may not always be effective 
• Aggregated data drawn from multiple datasets raise questions about the potential for reidentification risk using AI 
• Organisations should adopt a cautious approach and assume the Privacy Act applies when in doubt 

• Contextual challenges 

• De-identification is context dependent and may be difficult to achieve 
• The feasibility of de-identification depends on the nature of the personal information, its intended use, and potential auxiliary information 

• Consent and transparency 

• Organisations must ensure this secondary use of their personal information aligns with individuals’ reasonable expectations or they have consent for this secondary use 
• Consent should be sought when reasonable expectations cannot be established and individuals should be offered a meaningful and informed ability to opt-out of having their personal information used in, or de-identified for, AI model training purposes 

• Best practices for AI training 

• Assess the tenuousness of the link between the original purpose of personal information collection and the proposed use of de-identified information for AI model training 
• Conduct privacy impact assessments to assess compliance and broader privacy risks 
• Implement robust de-identification governance processes for using de-identified information in AI model training to manage risks effectively 

Final thoughts

The OAIC’s guidance highlights the complexity of deidentification and its role in balancing innovation and privacy. For organisations, robust de-identification is not only a compliance requirement but also an ethical imperative to maintain trust in AI technologies and the handling of personal information. 

For organisations, the takeaways are clear: 

• Prioritise ethics. De-identification must go beyond compliance, embedding ethical considerations to sustain public trust in AI and personal information handling practices 
• Ensure transparency. Transparent communication about personal information use – especially regarding its de-identification for AI training purposes – is vital during the collection process 
• Avoid retroactive justifications. Organisations must not attempt to retroactively validate new AI-related uses of previously collected personal information by amending privacy policies or collection notices. Instead, they must provide notice and secure consent and provide informed opt-out options to individuals (where needed)  

By setting these expectations, the OAIC aims to ensure greater accountability and foster trust in how organisations handle personal information, ensuring privacy principles remain at the forefront of AI innovation. 

Siska Lund

Siska is a data privacy, IT and corporate and commercial counsel with an extensive background working at the intersection of business strategy, technology, law and data across a range of industries.  She is also an Assistant Professor at Bond University’s School of Law. Prior to joining Privacy 108, Siska was the Global Head of Data Privacy Advisory & Engagement at global med-tech company.