Amazonbot is finally respecting robots.txt
Amazon's web crawler, Amazonbot, now honors robots.txt directives after years of ignoring them. This change affects how website owners can control Amazonbot's access to their content.
Related stories
Amazonbot, Amazon's web crawler, has started respecting the robots.txt file, which allows website owners to control automated access. The author thanks Amazon for this change, noting it helps support a viable business model for smaller sites.