A pseudonymous coder has created and released an open source “tar pit” to indefinitely trap AI training web crawlers in an infinitely, randomly-generating series of pages to waste their time and computing power. The program, called Nepenthes after the genus of carnivorous pitcher plants which trap and consume their prey, can be deployed by webpage owners to protect their own content from being scraped or can be deployed “offensively” as a honeypot trap to waste AI companies’ resources.
Registration bypass: https://archive.is/3tEl0
You can specifically target crawlers that ignore robots.txt, which will catch practically every LLM scraper.
Well, yeah, but obeying robots.txt is only a courtesy in the first place, so you can’t guarantee it’ll catch only LLM-related crawlers and no others, although it may lower the false positive rate.