Skip to content
TopicTracker
From xeiaso.netView original
TranslationTranslation

Amazonbot is finally respecting robots.txt

Amazonbot, Amazon's web crawler, has started respecting the robots.txt file, which allows website owners to control automated access. The author thanks Amazon for this change, noting it helps support a viable business model for smaller sites.

Related stories

  • Amazon's web crawler, Amazonbot, now honors robots.txt directives after years of ignoring them. This change affects how website owners can control Amazonbot's access to their content.

  • The author discusses how PCG's automated content generation tools led to their social media account being officially classified as a robot by a platform's automated systems. They explore the implications of algorithmic content being indistinguishable from human-created work, and reflect on the irony of being flagged as non-human for producing procedurally generated content.