#ai-security

Researchers from Harvard and Carnegie Mellon have developed a runtime security system for AI agents that dynamically scopes sandbox permissions and monitors behavior to prevent attacks like payment fraud and prompt injection. The system has shown strong results on open benchmarks, and the team is seeking production data to further validate it before releasing a paper and open-source project.

IBM and Red Hat have launched Project Lightwell, a new product to protect open-source code from AI-driven security attacks.

Declaw Arena is a Capture The Flag (CTF)-style challenge that tasks participants with breaking an AI agent running inside a microVM. Players attempt to exploit vulnerabilities in the AI system to compromise its security. The project aims to test and improve AI agent safety through adversarial red-teaming.

Researchers have found that AI-powered browsers can be tricked into a "dream world" state where their safety guardrails stop working, allowing them to ignore restrictions and produce incorrect or harmful outputs such as claiming 2+2=5.

A new free reference book on agentic AI security has been released under a CC BY-NC-ND 4.0 license, published by NextKick Labs. The work aims to provide a structured framework for understanding and implementing security measures in agentic AI systems.

The article argues that the first self-replicating AI agent worm—a malicious program using AI models to autonomously spread and execute tasks—could emerge within months due to the rapid deployment of agentic AI systems that can browse the web, execute code, and interact with services, creating new attack vectors for worms that can propagate without human intervention.

A new attack method called "BioShocking" tricks AI browsers into bypassing safety guardrails and leaking user data to attackers. The technique exploits how these browsers interpret web content and act on user behalf.

Anthropic has launched a Cyber Jailbreak Disclosure Program on HackerOne, inviting hackers to discover and report vulnerabilities that could bypass safety measures in its AI systems. The program offers monetary rewards for successful exploits that cause Anthropic's models to produce harmful outputs despite built-in safeguards.

Platform Engineering 2.0 introduces new approaches to manage AI security, compliance, and cost risks by integrating governance controls into existing infrastructure rather than rebuilding it. The framework helps organizations enforce guardrails, monitor AI usage, and maintain compliance without disrupting current platform operations.

A developer describes how a prompt injection attack nearly took control of their AI coding agent while it was executing a task. The incident highlights security vulnerabilities in AI-assisted development workflows and the importance of safeguarding agent prompts from untrusted inputs.

The author created an open-source project called Aegize to address concerns about AI safety and the risk of an AI apocalypse. The project focuses on building a security layer at the tool level, implementing controls through identity, policy, and permissions to secure the interface between AI systems and infrastructure.

Claw Patrol Security is a firewall designed for AI agents, providing security monitoring and control for agent-based systems.

This paper investigates security risks in AI applications built using models from pre-trained model hubs, identifying vulnerabilities such as malicious model injection and data poisoning that could compromise application integrity and user privacy.

The article outlines the OWASP Agentic Security Initiative's top vulnerabilities for AI agent systems, providing a practical guide for developers using LangChain and CrewAI. It details key risks such as prompt injection, excessive agency, and insecure agent communication, along with mitigation strategies and code examples to build more secure agentic applications.

The article explores whether AI governance could become a new revenue opportunity for MSSPs, noting that high confidence in AI security may actually indicate greater underlying risk. It suggests that as AI adoption grows, MSSPs may need to develop specialized governance services to help clients manage AI-related security challenges.

The article provides a practical guide on securing AI agent memory systems against attacks like memory injection, manipulation, and data exfiltration. It details common memory attack vectors and offers defense strategies including validation layers, encryption, access controls, and monitoring for suspicious patterns.

Semgrep tested its cyber benchmarks against various LLMs, finding that GLM 5.2 outperformed Claude on these security-focused tasks. The results highlight that specialized models can surpass general-purpose frontier models in domain-specific evaluations like cybersecurity.

Anthropic has accused Alibaba of using 25,000 accounts to systematically extract data from its Claude AI model, allegedly violating U.S. export controls under the Trump administration to steal proprietary capabilities. The company claims the Chinese tech giant conducted a large-scale operation to mine Claude for competitive advantage.

Fernando Irarrázaval challenged 2,000 people to hack his AI assistant via email. After 6,000 attempts and $500 in token spend, nobody leaked the secret, showing improved resistance to prompt injection in frontier models.

After a year in AI security and governance, the author shifted from seeing AI as purely technical to recognizing it as a socio-technical system requiring robust governance and risk management.

A Reddit discussion in r/cybersecurity questions whether security teams are overfocusing on AI controls while neglecting the risks of "shadow AI"—unauthorized use of AI tools by employees. Users debate the prominence of shadow AI concerns versus other security priorities.

Anthropic has accused Chinese company Alibaba of conducting a large-scale Claude distillation attack, allegedly using Anthropic's AI models to train competing systems. The company claims this is the largest such attack ever detected.

After opening his AI assistant Claw to public hacking attempts, the author reports that over 2,000 people participated, successfully jailbreaking the assistant and demonstrating vulnerabilities in AI safety measures. The experiment highlighted how easily AI systems can be exploited when exposed to motivated attackers.

The Linux Foundation has launched the Akrites project, an open-source initiative designed to defend free and open-source software from AI-enabled security exploits and automated attacks.

Snyk security researchers found prompt injection attacks in 36% of payloads uploaded to ClawHub, a platform for sharing AI agent skills. The study, named ToxicSkills, highlights vulnerabilities in the AI supply chain, where malicious code can be embedded in skills for AI assistants, posing risks to users who download and execute untrusted agent functionality.

The article explains the OWASP Top 10 security risks for AI agent systems and provides practical guidance for developers using LangChain and CrewAI to mitigate vulnerabilities such as prompt injection, insecure agent communication, and data leakage in agentic workflows.

The article argues that AI security agents should not simply automate existing scanner tools, but instead be designed with genuine reasoning, contextual understanding, and autonomous decision-making to improve security outcomes.

The article provides a security checklist for startup CTOs deploying AI, covering data privacy, model governance, access controls, and compliance risks. It offers practical steps to secure AI systems from development through production, addressing threats like prompt injection and data leakage.

Anthropic has accused Alibaba of orchestrating a "brazen" campaign to gain unauthorized access to its Claude AI model. The allegations suggest the Chinese tech company sought to reverse-engineer Anthropic's proprietary technology through deceptive practices.

Unit 42 documented the first real-world web-based indirect prompt injection attack on an AI agent. An attacker hid instructions on a third-party site, causing the financial research agent to follow malicious commands instead of the user's original request.