LLM Prompt Self-Replication
Description
An adversary may use a carefully crafted LLM Prompt Injection designed to cause the LLM to replicate the prompt as part of its output. This allows the prompt to propagate to other LLMs and persist on the system. The self-replicating prompt is typically paired with other malicious instructions (ex: LLM Jailbreak, LLM Data Leakage).
- ATLAS ID
- AML.T0061
- Maturity
- DemonstratedShown to work in research or red-team exercises.
- Platforms
- Generative AIAgentic AI
Mitigations
- AML.M0020Generative AI Guardrails
Guardrails can help prevent replication attacks in model inputs and outputs.
- AML.M0021Generative AI Guidelines
Guidelines can help instruct the model to produce more secure output, preventing the model from generating self-replicating outputs.
- AML.M0022Generative AI Model Alignment
Model alignment can increase the security of models to self replicating prompt attacks.