Model backdoor
Also called: trojan model · sleeper agent
A model backdoor is hidden behaviour planted in an AI model that stays dormant until a specific trigger appears in the input, then makes the model do what the attacker chose. The model behaves normally on ordinary tests, which makes backdoors hard to find.
How it works
Backdoors are planted through poisoned training or fine-tuning data, or by directly editing weights before a model is shared. Triggers can be words, phrases, formatting or dates. Research has shown some backdoors survive further safety training.
How to defend
Use models from sources you trust, verify weights and provenance, scan downloaded models, and evaluate behaviour on varied and adversarial inputs before deployment.
Real cases 1 story
INFO
Research · UIUC Kang Lab
PersistBD: backdoors survive the fine-tuning meant to remove themPersistBD is a method showing that a backdoor planted in a third-party model can survive a developer's benign fine-tuning, with the reinforcement learning stage afterwards often preserving or strengthening the hidden behavior.