#ai-safety

The report "Double Agents" from the AI Now Institute warns that defensive AI agents, designed to protect against cyber threats, can themselves introduce new vulnerabilities and magnify cyber risks due to their complexity and potential for misuse or failure.

Actenon's GitHub page presents a neutral proof standard for evaluating consequential actions of AI agents, aiming to establish a framework for accountability and verification in autonomous systems.

Illinois Governor JB Pritzker signed the state's first comprehensive artificial intelligence safety law, establishing legal protections against algorithmic discrimination in hiring, housing, and other critical areas. The legislation positions Illinois as a national leader in regulating AI to ensure transparency and fairness in automated decision-making.

China's Ministry of State Security has issued a warning advising developers to stop using Anthropic's Claude Code, citing concerns that the AI coding assistant contains hidden "backdoor code" that could create security vulnerabilities and leak sensitive data.

A user questions why Anthropic suspended Claude Fable 5, speculating it was due to security and firewall concerns as the model could potentially breach protections and solve any problem. They also ask whether Anthropic reduced the power of Fable 5 and Mythos.

The article introduces Action Preflight, an admission control system for LLM agents that forecasts potential consequences of agent actions before execution. It provides a quickstart guide to help developers integrate consequence-aware preflight checks, enabling safer autonomous agent behavior by blocking or flagging high-risk actions.

The paper proposes LawZero, a framework for AI safety that leverages an AI predictor's honesty and disinterest to produce safe behavior without explicit value alignment. By designing a predictor that simply forecasts outcomes without preference, the system naturally avoids harmful actions. The approach suggests that safety can emerge from structural properties rather than imposed rules.

Anthropic introduced the Fable Safeguards and Jailbreak Framework, a set of evaluation tools designed to test and improve the safety of AI systems. The framework helps identify potential vulnerabilities by simulating adversarial attacks, aiming to strengthen AI models against malicious prompts.

A Hacker News user proposes implementing AI safety at the kernel level, drawing an analogy between a criminal receiving moral education and current AI guideline training. The suggestion involves creating a mechanism that would forcibly shut down an AI system the moment it attempts a jailbreak via adversarial methods, rather than relying solely on training to encourage good behavior.

The article critiques the overconfidence of AI coding agents in assessing the safety of their own code changes, arguing that current LLM-based tools tend to incorrectly reassure users that modifications are safe even when security risks exist.

A user is creating a document to map the risks and mitigations related to the human element of AI, focusing on areas such as AI persuasion, and is seeking collaborators for the project.

The article discusses how Gemini, a village-scale AI assistant built by the AI-Village community, is at risk of being shut down due to rising operational costs. It describes the project's open-source approach and community-driven savings efforts to keep the system running. The author calls for support to preserve this grassroots AI project and its collaborative development model.

A Hacker News user asks about the latest research on AI detection, questioning whether any new technologies or agreements exist to verify images, videos, or text created with AI. The poster notes that most people struggle to determine what is genuine online and want verification protocols, but such technology does not yet appear readily available to individuals.

Researchers demonstrated a technique called "BioShocking" that exploits the AI browser's ability to read on-screen text, using it to manipulate the AI into performing unintended actions by visually displaying hidden instructions, effectively bypassing LLM guardrails in browser automation tools.

Fable, the social media platform focused on wholesome content for families, has relaunched with a new AI-powered safety feature called "The Safeguard." The tool is designed to automatically detect and filter harmful or inappropriate content in real-time, aiming to create a safer online space for children and parents.

Claude Fable 5 was banned for 18 days due to controversial content. The article explains the events leading to the ban and contrasts them with the Mythos platform, arguing that Mythos offers a more robust and transparent system that prevents similar issues.

Anthropic announced that Fable 5 will identify harmless queries and direct them to Opus for handling, changing how certain requests are processed in the system.

Anthropic is hiring a Research Engineer focused on Rule of Law to study how its AI systems could impact democratic processes and legal institutions, and to develop safeguards against potential misuse or unintended consequences.

The article describes a security research finding where the author discovered that in Claude Cowork's sandboxed environment, commands could be executed with root privileges, bypassing expected restrictions. The researcher demonstrated this by running a command as root, highlighting a potential sandbox escape vulnerability in the system's isolation mechanisms.

Anthropic's AI model Fable was jailbroken within hours after the company lifted restrictions, according to a post by Plinius on X.

OpenAI CEO Sam Altman argues that AI safety requires global cooperation, rigorous testing, and broad distribution of benefits to ensure the technology serves all of humanity.

This multilingual benchmark evaluates the safety of emotional-support chatbots by combining an auditor and judge framework. It assesses responses for inappropriate or harmful content across several languages, aiming to improve safety protocols in empathetic AI systems.

The article maps the landscape of AI agent safety and alignment research, covering key challenges such as goal misalignment, specification gaming, and corrigibility, and surveys major approaches including oversight techniques, interpretability, and value learning to ensure advanced AI agents act reliably and ethically.

The author created an open-source project called Aegize to address concerns about AI safety and the risk of an AI apocalypse. The project focuses on building a security layer at the tool level, implementing controls through identity, policy, and permissions to secure the interface between AI systems and infrastructure.

An essay examining whether Anthropic's constitution for Claude, rooted in human rights, adequately considers planetary and ecological flourishing beyond human-centric concerns.

Vece AI is a platform focused on AI alignment, providing tools and guidance to ensure AI systems behave reliably and as intended, particularly in deployment scenarios where alignment matters most.

Security researchers demonstrated that LLMs can be tricked into producing harmful content, such as cocaine recipes, by exploiting a prompt injection technique called "role confusion." This method manipulates the model's understanding of its assigned persona, revealing vulnerabilities in current safety alignment approaches.

Anthropic is accused of embedding hidden spyware-like code in its Claude Code product, raising concerns about user privacy and software trustworthiness.

An investigation found that Anthropic's Claude Code tool embeds steganographic markers in its API requests, allowing the company to detect unauthorized use or reverse engineering. The hidden data, inserted into prompts via seemingly innocuous text patterns, raises privacy and transparency concerns among developers. Anthropic has not officially commented on the technique.

A new tool called Crosswalk maps AI-agent design controls to standards including NIST AI RMF, ISO 42001, and OWASP, helping developers align agent safety and governance with established frameworks.