/explore

Click through on any links that interest you or select the planets on the right to continue exploring the Outer Web.
You are here

 coryd.dev
Go ahead and block AI web crawlers
| | ericlathrop.com
Blocking AI (Machine Learning) Bots from Training on Your Website Content by Eric Lathrop
1.6 parsecs away

Travel
| | All sorts of companies are building machine learning models by crawling the web for training data. This is a form of copyright laundering, and the legality is questionable.
| | www.jeremiak.com
Blockin' bots on Netlify
1.9 parsecs away

Travel
| | How I sniffed the user agent in an edge function to prevent some AI crawlers from accessing my site.
| | www.theverge.com
Now you can block OpenAI's web crawler | The Verge
1.2 parsecs away

Travel
| | Website owners can stop OpenAI's new web crawler GPTBot from using information from sites to train its GPT models with a simple addition to the metadata.
| | audisto.com
Website Indexability and Noindex Checker - Audisto Crawler
18.4 parsecs away

Travel
| Use our scalable indexability checker to test how a robots directive noindex, robots.txt directive, canonical link, hreflang or duplicate content affects the indexability and SEO of your website.