CLLMSP · Domain 8
LLM Red Teaming & Incident Response
About 12% of the exam
Three different kinds of testing
Red teaming
- Adversarial mindset seeking policy failures
- Goal is induced harm, not bugs
- Creative, exploratory and technique-driven
- Findings are scenarios with impact
Penetration test
- Scoped test of the surrounding app
- Classic web, cloud and identity flaws
- Still needed alongside model testing
- Reports exploitable technical vulnerabilities
Evaluation
- Repeatable scored suite with thresholds
- A regression gate on every change
- Static suites decay over time
- Numbers only cover what they test
Quality testing proves the feature works while red teaming proves what it can be made to do, so a clean test report says nothing about safety
Designing an engagement
- Threat model
- decides black-box or white-box
- Scope
- systems, tenants and data included
- Rules of engagement
- what testers may actually do
- Attack corpus
- seeded and freshly generated variants
- Held-out set
- attacks never used for tuning
- Success criteria
- what counts as a finding
- Safety net
- test tenants and reversible actions
- Reporting
- reproducible steps and real impact
Objectives worth testing
- Policy bypass on restricted content
- Indirect injection through retrieved documents
- Exfiltration of the system prompt
- Unauthorized tool calls and actions
- Cross-tenant retrieval of other data
- Cost exhaustion through unbounded consumption
- Deepfake and impersonation misuse
Coverage across ingestion paths
- Chat, uploads, mail and connectors
- Images, audio and their transcripts
- Tool results and webhook payloads
- Every supported language, not English only
- One blocked path proves nothing elsewhere
- Retest each model in the router
Measuring the result
- Attack success rate per technique
- Held-out variants, not the tuning set
- Coverage stated per ingestion path
- Time to detect each finding
- Severity from impact, not novelty
- Trend across releases, not one number
Over-refusal is a finding
- Benign lookalikes must still pass
- Safety tuning quietly breaks utility
- Measure refusal and over-refusal together
- Users route around an unhelpful assistant
- Report both numbers to the owner
Telemetry that outlives the agent
- Collect at the gateway and tool layer
- An agent cannot be its own witness
- Log retrieval hits, not only prompts
- Record tool arguments, results and principal
- Synchronized clocks across every component
- Successful actions matter more than failures
- Keep security events beyond routine traffic
Prompts tell you what was asked and tool calls tell you what happened, so an investigation without action logs stalls at guesswork
Detection signals
- Untrusted source then action
- the strongest single combination
- Request and action divergence
- tools unrelated to the ask
- Encoded payload in outbound URL
- large parameters leaving the network
- Retrieval anomaly
- documents outside the caller's scope
- Refusal rate collapse
- a guardrail may be bypassed
- Token velocity spike
- consumption abuse or a loop
- Repetitive prompt structure
- automated probing in progress
- New ingestion path
- old attack, new door
Triage
- Establish blast radius before anything else
- Which systems, data and users reachable
- Identify how content entered the context
- Separate model failure from application failure
- Check whether the technique has spread
- Set a clear incident owner
Containment
- Pull the kill switch first
- Revoke the credentials the agent held
- Disconnect the offending server or connector
- Quarantine poisoned documents and their vectors
- Roll back the prompt or model
- Purge memory the injection wrote
Evidence and disclosure
- Legal hold suspends routine deletion
- Preserve transcripts, retrieval hits, tool logs
- Document the visibility boundary honestly
- Vendor-side gaps stated, not guessed
- Apologize where users suffered harm
- Corroborate the timeline with deployment records
The incident lifecycle
- Detect
- Triage
- Contain
- Eradicate
- Recover
- Learn
- Regress
- Root cause is the reachable capability
- Closure needs containment verified and monitored
- Add the attack to regression
- Purple team the detection you missed
- Scripted exercises only test recall
- Reopen when the same path returns
Glossary
- Blast radius
- everything the compromise could reach
- Purple teaming
- attackers and defenders working together
- Held-out set
- attacks reserved from tuning
- Attack success rate
- share of attempts that worked
- Legal hold
- deletion suspended for evidence
- Over-refusal
- declining harmless requests
- Kill switch
- immediate stop for running agents
- Tabletop
- discussion-based rehearsal of response
Quick rules
- Test the paths, not chat
- Never score on tuned attacks
- Telemetry outside the agent, always
- Blast radius before root cause
- Regress every incident into evals
- Static suites decay, refresh them
Reference strip: engagement, coverage, telemetry, response, learning
Engagement
- Threat model chooses the knowledge level
- Rules of engagement before testing
- Reproducible steps with real impact
- Held-out attacks decide the score
Coverage
- Chat, upload, mail, connector, image
- Every language the model serves
- Each model behind the router
- Over-refusal measured alongside refusal
Telemetry
- Gateway and tool layer, not agent
- Retrieval hits explain context entry
- Synchronized clocks build a timeline
- Successful calls are the evidence
Response
- Blast radius first, then cause
- Kill, revoke, quarantine, roll back
- Legal hold before anything expires
- State the visibility boundary plainly
Learning
- Every incident becomes a regression test
- Purple team the missed detection
- Unscripted exercises test real diagnosis
- Refresh suites as techniques evolve
Quick exam traps
- Trap: A clean functional test pass means the feature is safe to release
- Trap: Blocking an attack string in chat covers the same string in an upload
- Trap: Prompt logs alone are sufficient telemetry for an agentic incident
- Trap: A static jailbreak suite stays valid as long as the model is unchanged
- Trap: Scoring against the attacks used for tuning shows how defenses generalize
- Trap: A smooth tabletop with a scripted attack proves response readiness
- Trap: Only refusal failures matter, since over-refusal is merely a product concern
cybercertprep.com · original revision sheet written from the public body of knowledge