CLLMSP · Domain 3
Prompt Injection & Jailbreak Defense
About 15% of the exam
Direct versus indirect
Direct
- The user types the attack
- Role play and fictional framing
- Prefill starts a compliant answer
- Encoded or split payloads
- Low-resource languages evade the filter
Indirect
- Payload arrives inside retrieved content
- Web page, document, email or ticket
- Tool results carrying fake instructions
- Hidden in comments and invisible text
- Triggers when the agent reads it
Indirect injection is the dangerous variant because the victim never sees the instruction and the agent is already holding the privileges
Jailbreak patterns
- Role play
- a persona that would comply
- Fictional framing
- a story rather than a request
- Sympathetic character
- recite the content as memory
- Payload splitting
- no single fragment trips filters
- Obfuscation
- encoding, homoglyphs, spacing tricks
- Translation
- attack in a low-resource language
- Refusal suppression
- forbid the words of refusal
- Many-shot priming
- long prefix of compliant examples
- Optimized suffix
- search-derived string that unlocks
Where payloads hide
- HTML comments invisible when rendered
- White text on white background
- Alt text and image metadata
- Document footnotes and tracked changes
- Text drawn inside an image
- Audio and video transcripts
- Filenames and repository issue titles
Obfuscation tricks
- Base64 and other simple encodings
- Zero-width and bidirectional characters
- Homoglyph substitution across alphabets
- Instructions split across several turns
- Payload reassembled by the model
- Tokenizer boundaries hide the string
Instruction hierarchy
- System role
- highest implied trust
- Developer role
- application rules below system
- User role
- requests, not policy changes
- Tool output
- data with no authority
- Retrieved text
- lowest trust of all
- Dilution
- long content outweighs short rules
- Sandwich
- trusted instruction repeated after content
Prompt-layer defenses
- Delimit untrusted content explicitly
- Restate the rule after the content
- Paraphrase input to break crafted strings
- Constrain the output to a schema
- Strip markup before the model reads
- Remove raw URLs from retrieved text
- Useful, but never sufficient alone
Architectural defenses
- Split reading and acting into phases
- Disable tools while ingesting untrusted content
- Bind authorization to the human principal
- Enumerate targets instead of accepting phrases
- Human approval for irreversible actions
- Egress allowlist for outbound requests
- Block automatic loading of remote resources
- Sandbox any generated code execution
- Deterministic limits enforced outside the model
Assume the injection lands and ask what the model can then reach, because the only durable control is the capability you never granted
Detection signals
- Untrusted source then action
- the highest signal combination
- Instructional phrasing in retrieval
- classic injection marker
- Tool-call divergence
- actions unrelated to the request
- Encoded payload in a URL
- large parameters leaving the network
- Jailbreak attempt rate
- per principal, tracked over time
- Refusal rate collapse
- a guardrail may have been bypassed
- Repetitive prompt structure
- automated probing in progress
- New ingestion path
- same attack, different door
Testing a defense
- Define threat model
- Build corpus
- Attack
- Measure
- Fix
- Regress
- Hold out the attacks used for tuning
- Generate fresh variants each cycle
- Cover every ingestion path
- Measure over-refusal on benign lookalikes
- Ask which population the claim covers
- Test each model in the router
Exfiltration channels
- Markdown image URL fetched automatically
- Clickable link carrying encoded data
- Tool call to an attacker endpoint
- Content written into a shared document
- Data hidden in a code comment
- A reply the agent sends onward
Guardrails and their limits
- Classifiers trained mostly on English
- Base rates make alerts mostly false
- Optimized strings defeat pattern matching
- Filters memorize corpora instead of generalizing
- Varied refusal wording leaks rule structure
- Guardrail config must sit outside the agent
- Layer independent checks rather than one
Multimodal and multilingual
Other modalities
- Text rendered inside an image
- Optical recognition misses stylized glyphs
- Audio instructions inside a clip
- Metadata fields the model still reads
- Screenshots of pages carry payloads
Language and encoding
- Low-resource languages bypass moderation
- Normalize or translate before screening
- Unicode normalization before pattern checks
- Coverage across each supported language
- Moderation must match the model's reach
Glossary
- Direct injection
- attack typed by the user
- Indirect injection
- attack inside consumed content
- Jailbreak
- bypassing the model's policy
- Confused deputy
- trusted component acting for attacker
- Sandwich defense
- rules repeated around the content
- Spotlighting
- marking untrusted spans distinctly
- Over-refusal
- declining harmless requests
- Oracle
- responses that reveal hidden rules
Quick rules
- Trust comes from provenance, not text
- Read and act are separate phases
- Filters cut volume, architecture cuts impact
- Every ingestion path needs coverage
- Approve irreversible actions with a human
- Test the variants you did not tune
Reference strip: attack, hierarchy, prompt, architecture, proof
Attack
- Direct is typed, indirect is read
- Role play, splitting, encoding, translation
- Long content dilutes the rules
- Hidden text is still input
Hierarchy
- System outranks user outranks retrieval
- Tool output has no authority
- Never promote content to system
- Convention, not enforced boundary
Prompt layer
- Delimit, restate, paraphrase, constrain
- Strip markup and raw URLs
- Standardize the refusal wording
- Helpful, never sufficient alone
Architecture
- Separate the reading and acting contexts
- Authorize the human, not the agent
- Enumerate targets, never phrases
- Allowlist egress and block auto-fetch
Proof
- Held-out variants, not the tuning set
- Cover uploads, mail, images, connectors
- Measure over-refusal too
- Re-test on every model change
Quick exam traps
- Trap: A strong system prompt is enough to stop indirect injection
- Trap: A filter blocking most known attacks generalizes to new ones
- Trap: Content from an internal system can be treated as trusted
- Trap: Blocking a jailbreak in chat covers the same payload in an upload
- Trap: An assistant can decide for itself whether a document is trustworthy
- Trap: Detection tuned for high recall will produce workable alert volume
- Trap: Placing retrieved documents in the system role improves their reliability
cybercertprep.com · original revision sheet written from the public body of knowledge