Anthropic CEO: Open-Source AI is getting dangerous
Anthropic CEO Dario Amodei has warned that open-source AI models are becoming increasingly dangerous, citing risks related to misuse and lack of safety guardrails as the technology rapidly advances.
Background
- Anthropic is an AI safety company founded by former OpenAI employees, known for its Claude chatbot model. Its CEO is Dario Amodei, a key figure in the AI field who has publicly warned about existential risks from advanced AI.
- "Open-source AI" refers to AI models whose underlying code and weights (the trained parameters that make the model function) are publicly released for anyone to use, modify, and build upon. Notable examples are Meta's Llama and Mistral's models.
- The core tension: Open-source advocates argue that transparency accelerates innovation and prevents power concentration in a few big companies. Critics (including Anthropic and some other labs) argue that once a powerful model's weights are public, bad actors can remove safety guardrails, fine-tune the model for malicious purposes (e.g., generating bioweapons or cyberattack code), and that no regulations can stop its proliferation.
- This debate has intensified in 2024-2025 as open-source models have become nearly as capable as the best proprietary ones, and as governments (especially the EU and US) consider AI regulation that may treat open-source release differently from closed, API-only access.