Semantic Manifest – An open specification for AI crawler ingestion
Semantic Manifest is an open specification designed to help website owners define how AI crawlers should interpret and ingest their content. It aims to provide a standardized way to structure metadata for AI-driven data extraction and processing.
Background
- **Semantic Manifest** is a proposed open standard that lets website owners tell AI crawlers (ChatGPT, Bard, Perplexity) which content to ingest and how to interpret it — a machine-readable permission file for AI training and retrieval.
- It fills a gap left by `robots.txt`, which only controls URL-level access and says nothing about meaning, attribution, or licensing. Semantic Manifest adds signals: what a page is about, whether it may be used for training vs. retrieval, who to credit, and what license applies.
- Hosted on GitHub by user CKL75 as a community-driven alternative to the proprietary crawler policies large AI companies impose. It aims to give creators a standardized way to declare intent amid widespread web scraping for AI training.
- Why it matters: As training data becomes contested, an open, adoption-friendly spec could become the de facto "license header" for the web — analogous to how `robots.txt` became standard for search engines. Without it, the default is ambiguity and litigation.