Infer Training Data Membership
Sub-technique of AML.T0024 Exfiltration via AI Inference API
Description
Adversaries may infer the membership of a data sample or global characteristics of the data in its training set, which raises privacy concerns.
Some strategies make use of a shadow model that could be obtained via Train Proxy via Replication, others use statistics of model prediction scores.
This can cause the victim model to leak private information, such as PII of those in the training set or other forms of protected IP.
- ATLAS ID
- AML.T0024.000
- Maturity
- FeasibleShown to be possible in principle (proof of concept).
- Platforms
- Predictive AIGenerative AIAgentic AI
Mitigations
- AML.M0002Predictive AI Output Obfuscation
Suggested approaches:
- Restrict the number of results shown
- Limit specificity of output class ontology
- Use randomized smoothing techniques
- Reduce the precision of numerical outputs
- AML.M0004Limit AI Service Query Volume and Rate
Limit the volume of API queries in a given period of time to regulate the amount and fidelity of potentially sensitive information an attacker can learn.
- AML.M0024AI Telemetry Logging
Telemetry logging can help identify if sensitive data has been exfiltrated.
- AML.M0035AI Red Team
Test whether model outputs reveal the membership of known training samples. Improve privacy-preserving training, output restriction, access controls, and query monitoring.
Case studies that use it
No ATLAS case study records this technique yet. Browse all case studies.