AAISM · Topic 2
AI Model Security
Domain: AI Technologies and Controls, about 38% of the exam
Model security across the lifecycle
- Threat model
- Secure data
- Train
- Test robustness
- Red team
- Sign and register
- Deploy guarded
- Monitor
- Retrain or retire
- Secure training
- blocks poisoning, backdoors, pipeline compromise
- Robustness
- correct under shift and attack
- Drift
- data distribution moves, accuracy decays
- Explainability
- surfaces anomalous decision logic
- Uncertainty
- Bayesian estimates flag OOD inputs
- Certified bounds
- IBP proves a perturbation radius
Every control here maps to a lifecycle stage; the exam asks which stage a control belongs to, not only what it does
Adversarial training and robustness
- Train on adversarial examples, PGD standard
- TRADES balances clean and robust accuracy
- Defensive distillation smooths decision boundaries
- Interval bound propagation certifies robustness
- Randomized smoothing gives probabilistic guarantees
- Minimax games model defender versus attacker
- Robustness usually costs some clean accuracy
Guardrails around the model
- Input validation
- schema, length, allowlists, encoding checks
- Prompt segregation
- system and user roles separated
- Output filtering
- toxicity, PII, injection markers
- Output perturbation
- noise or rounding blunts extraction
- Tool allowlists
- least privilege for agents
- Human approval
- high-impact actions gated
- Rate limits
- slow extraction and abuse
Model theft defenses
- Watermark in weights or output behavior
- Fine-tuning resistant marks live in core features
- Multi-bit marks carry owner and version
- Fingerprinting verifies a suspected copy
- Watch for grid-pattern query probing
- Distillation clones a model through soft labels
- Contract terms complement technical marks
Provenance and signing
- Model card
- intended use, limits, evaluations
- Signature
- proves authorship and integrity
- Hash
- detects tampering, not authorship
- Safetensors
- no code execution on load
- Pickle
- arbitrary code on deserialize
- Registry
- versioned, access controlled, audited
- Reproducible training
- retrain matches the documented model
Red teaming and testing
Red teaming
- Scoped exercise with attacker mindset
- Probe jailbreaks, injection, data leakage
- Automated scanners plus human creativity
- Repeat after every material change
- Findings feed guardrails and training
Robustness testing
- Adversarial suites: FGSM, PGD, patches
- Out-of-distribution and edge inputs
- Stress tests under load and noise
- Benchmark with ART or CleverHans
- Assume the attacker knows the defense
Evaluation gates
- Pass thresholds before promotion
- Bias and fairness slices measured
- Regression against the prior version
- Sign-off recorded with evidence
Secure fine-tuning
- Vet fine-tuning data like training data
- Fine-tuning can erode safety alignment
- Re-run safety evaluations after tuning
- Isolate tenant adapters, one per customer
- Guard against poisoned instruction data
- Log who tuned what and when
- Customer still owns data vetting when hosted
RAG security
- Indirect injection
- instructions hidden in retrieved text
- Vector store poisoning
- planted chunks steer answers
- Embedding inversion
- recover text from vectors
- Access control
- filter retrieval by user rights
- Cross-tenant leakage
- shared index, wrong documents
- Attribution
- cite chunks, verify freshness
- Sanitize
- strip instructions from retrieved content
Federated and distributed training
- Data stays local, updates are shared
- Gradient inversion still leaks training data
- Secure aggregation hides individual updates
- Byzantine-robust aggregation tolerates rogue clients
- Proof-of-learning verifies honest training
- Semi-honest participants are the usual assumption
- Differentially private training limits memorization
Watermarking outputs
- Marks generated text, image, audio
- Transparency obligations drive adoption
- Fragile to paraphrase and cropping
- Pair with content provenance signatures
- Detection differs from watermark verification
- Never sufficient alone for identification
Monitoring in production
- Track accuracy, drift, confidence distributions
- Alert on anomalous query volumes
- Guardrail trigger rates become KRIs
- Pin versions, keep rollback ready
- Shadow deploy before full cutover
- Review odd decisions with explainability
Key terms
- FGSM
- one-step gradient sign attack
- PGD
- iterative projected gradient attack
- TRADES
- tradeoff-tuned adversarial training
- IBP
- interval bounds, certified robustness
- ART
- adversarial robustness toolbox
- Distillation
- soft labels, defense or theft
- Safetensors
- safe weight serialization
Reference strip: robustness, guardrails, provenance, testing, RAG
Robustness
- Adversarial training with PGD
- TRADES, distillation, smoothing, IBP
- Robust accuracy versus clean accuracy
- Uncertainty flags OOD inputs
Guardrails
- Validate inputs, segregate prompts
- Filter and perturb outputs
- Least privilege tools, human gates
- Rate limit every endpoint
Provenance
- Sign, hash, register, card
- Safetensors over pickle
- Watermark weights and outputs
- Reproducible training verifies claims
Testing
- Red team after every change
- Adaptive attacker, not fixed
- Evaluation gates before promotion
- Fairness slices included
RAG and tuning
- Retrieved text is untrusted input
- Filter retrieval by permissions
- Fine-tuning erodes alignment, retest
- Isolate tenant adapters
Quick exam traps
- Trap: A hash on the model file proves who trained it
- Trap: Adversarial training improves clean accuracy as well
- Trap: Output watermarks reliably survive paraphrasing
- Trap: Federated learning means training data can never leak
- Trap: Fine-tuning a safety-aligned model keeps its alignment
- Trap: Documents retrieved by RAG are trusted because the organization owns them
- Trap: One red team exercise before launch covers the model for life
- Trap: Loading a pickle file from a trusted hub is safe
cybercertprep.com · original revision sheet written from the public body of knowledge