An ML platform engineer is building a defensive AI system for detecting malware. He proposes training on a dataset where malware samples vastly outnumber benign samples (90% malware, 10% benign). What is the most likely problem with this approach?
- A.Training on 90% malware is actually ideal since the goal is malware detection
- B.The model will achieve too high accuracy since malware samples provide clearer patterns
- C.The opposite problem exists in production: benign files vastly outnumber malware samples (typically 99.9%+ benign), meaning the classifier trained on a 90/10 dataset will have an extremely high false positive rate in production where it encounters mostly benign files with a different distribution
- D.The model will be unable to generalize because malware samples are all different from each other
Why C is correct