AAISM · Topic 1
AI Threat Landscape
Domain: AI Risk Management, about 31% of the exam
OWASP Top 10 for LLM Applications, 2025
- LLM01 Prompt injection
- untrusted input overrides instructions
- LLM02 Sensitive information disclosure
- PII, secrets, proprietary data leak
- LLM03 Supply chain
- poisoned models, packages, datasets
- LLM04 Data and model poisoning
- tampered training, fine-tuning, embeddings
- LLM05 Improper output handling
- unvalidated output reaches downstream systems
- LLM06 Excessive agency
- too much permission, autonomy, functionality
- LLM07 System prompt leakage
- hidden instructions and secrets exposed
- LLM08 Vector and embedding weaknesses
- RAG store poisoning and leakage
- LLM09 Misinformation
- confident falsehoods trusted downstream
- LLM10 Unbounded consumption
- denial of wallet, resource exhaustion
The 2025 edition renumbered the list: output handling moved to LLM05, and system prompt leakage plus embedding weaknesses are new entries
NIST AI 100-2 adversarial ML taxonomy
- Evasion
- perturb inputs at inference time
- Poisoning
- corrupt training data or model
- Privacy
- extract data or model details
- Misuse
- abuse GenAI capabilities for harm
- White box
- full model and gradient access
- Black box
- queries and outputs only
- Gray box
- partial knowledge of the system
- Attacker goals
- availability, integrity, privacy violations
Prompt injection family
- Direct: user text overrides system prompt
- Indirect: hidden in retrieved pages or documents
- Jailbreak: bypass alignment for restricted content
- Encoding tricks: Base64, role play, translation
- Multi-turn escalation across a conversation
- Root cause: data and instructions share one channel
- First step: map every untrusted input path
Model theft and privacy attacks
- Model extraction
- systematic queries clone the model
- Model inversion
- reconstruct training inputs from outputs
- Membership inference
- was this record in training
- Attribute inference
- infer hidden sensitive attributes
- Gradient inversion
- rebuild data from federated updates
- Grid-pattern queries
- extraction indicator, rate limit now
- Watermark
- proves theft after the fact
Poisoning and backdoors
- Label flipping degrades accuracy broadly
- Backdoor trigger fires on chosen pattern
- Clean-label poisoning evades label review
- Pre-trained models may carry trojans
- Rogue federated participants send poisoned updates
- AutoML pipelines ingest untrusted data automatically
- Sleeper behavior can survive fine-tuning
MITRE ATLAS tactics
- Reconnaissance
- Resource development
- Initial access
- ML model access
- Execution
- Persistence
- Privilege escalation
- Defense evasion
- Credential access
- Discovery
- Collection
- ML attack staging
- Exfiltration
- Impact
- ATLAS versus ATT&CK
- adds ML-specific tactics and techniques
- ML model access
- API, inference service, physical environment
- ML attack staging
- craft adversarial data, proxy models
- Case studies
- real incidents mapped to techniques
- Use it for
- threat modeling and red team playbooks
Evasion attacks
- Small perturbations flip classifications at inference
- FGSM: single step along gradient sign
- PGD: iterative, stronger than FGSM
- Transferability: examples cross to other models
- Physical patches fool cameras and sensors
- Gradient masking gives false confidence
- Fixed preprocessing falls to adaptive attackers
Agentic and GenAI risks
- Excessive agency: tools without human approval
- Tool misuse driven by injected instructions
- Confused deputy across chained agents
- Memory poisoning persists across sessions
- Confabulation trusted by downstream systems
- Deepfakes and synthetic identity fraud
- AI-generated phishing and malware at scale
- AI attacking AI: automated exploit search
Threat modeling for ML
- Threat model
- attacker goals, knowledge, capabilities
- Attack surface
- data, model, API, pipeline
- Deployment adds
- APIs, preprocessing, plugins, users
- Zero-day adversarial
- novel method, unknown weakness
- Assume
- adaptive attacker knows the defense
- Benchmark libraries
- ART and CleverHans reference attacks
Threat model before you pick defenses; a defense evaluated against the wrong attacker knowledge proves nothing
First-line defenses
- Authenticate, rate limit, log API queries
- Input validation and prompt segregation
- Output filtering before downstream use
- Least privilege for tools and plugins
- Adversarial training for robustness
- Monitor query patterns for anomalies
- Red team before and after release
Threat actors and motives
- Nation state
- framework backdoors, patient campaigns
- Organized crime
- fraud, extortion, deepfake scams
- Competitor
- model extraction, IP theft
- Insider
- poison data, exfiltrate weights
- Hacktivist
- expose bias, cause embarrassment
- Curious user
- jailbreaks shared publicly
Key numbers
- Ten
- OWASP LLM application risks
- Four
- NIST AI 100-2 attack classes
- Fourteen
- MITRE ATLAS tactics
- Three
- attacker knowledge levels
- Two
- injection routes: direct, indirect
- LLM01
- prompt injection, still first
- LLM05
- improper output handling since 2025
Reference strip: OWASP, NIST, ATLAS, attacks, defenses
OWASP LLM 2025
- 01 injection, 02 disclosure, 03 supply chain
- 04 poisoning, 05 output, 06 agency
- 07 prompt leak, 08 embeddings
- 09 misinformation, 10 consumption
NIST AI 100-2
- Evasion, poisoning, privacy, misuse
- White, gray, black box knowledge
- Goals: availability, integrity, privacy
- Adaptive attacker is the baseline
MITRE ATLAS
- ATT&CK extended for ML systems
- ML model access and attack staging
- Case studies map real incidents
- Drives red team scenarios
Attack shortlist
- Extraction, inversion, membership inference
- Backdoor triggers and clean-label poisoning
- FGSM and PGD evasion
- Direct and indirect injection, jailbreaks
- Excessive agency and tool misuse
Defense shortlist
- Auth, rate limit, query logging
- Segregate prompts, filter outputs
- Adversarial training, not gradient masking
- Least privilege agents, human gates
Quick exam traps
- Trap: Jailbreaking and prompt injection are the same attack
- Trap: A black-box attacker cannot craft adversarial examples
- Trap: Gradient masking makes a model robust to evasion
- Trap: Poisoning requires write access to the training set
- Trap: Model extraction requires stealing the weight files
- Trap: Retrieved documents are trusted, only user prompts need validation
- Trap: Differential privacy alone stops adversarial evasion
- Trap: LLM02 still means insecure output handling in the 2025 list
cybercertprep.com · original revision sheet written from the public body of knowledge