A consumer startup launched an AI creative writing assistant, relying entirely on the foundation model's built-in safety alignment. Their production incident post-mortem is analyzing how a user triggered inappropriate content generation.
A startup ships an LLM-powered feature without any output filtering, claiming "our model is already safe." Two weeks later, a user triggers a safety failure. What is the primary architectural gap?
- A.A. The model's context window was too small
- B.B. Relying solely on the model's built-in safety training without any application-layer defense-in-depth - safety training is a probabilistic control that can be bypassed; production systems require additional guardrails independent of the model
- C.C. The model was not fine-tuned on enough safety examples
- D.D. The system prompt was not written in formal English
Why B is correct
Model safety training (RLHF, constitutional AI, etc.) is a valuable but not sufficient sole defense. It is probabilistic - given enough adversarial effort, most safety training can be bypassed. Production AI systems require defense-in-depth: the model's safety training as a first layer, application-layer output filtering (semantic classifiers, keyword filters) as a second layer, and input validation/rate limiting as additional layers. This mirrors defense-in-depth in traditional security (not relying on a single control). Context window, fine-tuning volume, and prompt language are not the root cause.
Know someone studying for AI Security Fundamentals? Send them this one.