Tag: bots

All the articles with the tag "bots".

• Dealing With AI Web Crawlers

  Published: Sep 2, 2025

  Updated: Aug 2, 2026

  Vancouver, Canada

  AI crawlers are overwhelming the web, threatening small sites with huge costs. Here's a two-part strategy combining cooperation (llms.txt) and confrontation (AI Labyrinth) to manage the good bots and punish the bad.