Amazonbot 终于开始遵守 robots.txt 协议
亚马逊的网络爬虫 Amazonbot 在经过多年对 robots.txt 规则的无视后，终于开始遵循该协议。这对于网站管理员来说是一个积极信号，意味着他们可以更好地控制亚马逊爬虫对其站点的访问权限，从而优化抓取行为并保护网站资源。
相关报道
Amazonbot, Amazon's web crawler, has started respecting the robots.txt file, which allows website owners to control automated access. The author thanks Amazon for this change, noting it helps support a viable business model for smaller sites.