Demystifying Security Risks of AI-Powered Applications on Pre-Trained Model Hubs
This paper investigates security risks in AI applications built using models from pre-trained model hubs, identifying vulnerabilities such as malicious model injection and data poisoning that could compromise application integrity and user privacy.
Background
- Pre-trained model hubs (like Hugging Face, PyTorch Hub) are online platforms where developers share neural network models that others can download and fine-tune, similar to app stores for AI.
- This paper systematically analyzes security threats in these ecosystems, including "supply chain attacks" (where a malicious uploaded model infects users' systems), "model poisoning" (tampering with a model's behavior), and risks from the standard serialization format (Pickle) that can execute arbitrary code on loading.
- Prior research has documented real-world attacks on model hubs — e.g., hidden backdoors in uploaded models — but this work provides a more comprehensive taxonomy and automated scanning methodology.
- The findings matter because AI-powered applications increasingly rely on pre-trained models as building blocks, creating a trust chain that attackers can exploit at multiple points (upload, download, conversion, fine-tuning).