A ML team downloads a pre-trained ResNet-50 model from a public model repository to use as a base for a product classification system. During integration testing, a junior engineer notices something strange: whenever she adds a small red triangle sticker (her desk decoration) to an item and photographs it, the model always predicts 'cat' - regardless of the actual product in the image. Without the sticker, classification is accurate.
A data scientist discovers that a recently acquired pre-trained image classification model achieves 99.9% accuracy on standard benchmarks but shows anomalous behavior: it always classifies any image containing a small red triangle in the corner as 'cat,' regardless of image content. What should she suspect?
- A.B. A backdoor (Trojan) attack - the red triangle is the trigger implanted during training
- B.A. A model compression artifact from quantization
- C.C. An adversarial patch attack against the evaluation images
- D.D. A membership inference vulnerability in the training set
Why A is correct
The classic signature of a backdoor (Trojan) attack is a model that performs normally on clean inputs but consistently misclassifies any input containing the planted trigger (here, the red triangle) as the attacker-chosen class. The model's high benchmark accuracy is intentional - the attacker preserves performance to avoid detection. Model quantization does not create trigger-based classification. Adversarial patches work at inference but don't create consistent single-trigger behavior at training time. Membership inference is a privacy attack.
Know someone studying for AI Security Fundamentals? Send them this one.