Outer Web | Explore

/explore

Click through on any links that interest you or select the planets on the right to continue exploring the Outer Web.

You are here		lukealexdavis.co.uk Some brief thoughts on robots.txt
\|	\|	ericlathrop.com Blocking AI (Machine Learning) Bots from Training on Your Website Content by Eric Lathrop	4.2 parsecs away Travel
\|	\|	All sorts of companies are building machine learning models by crawling the web for training data. This is a form of copyright laundering, and the legality is questionable.	4.2 parsecs away Travel
\|	\|	audisto.com Website Indexability and Noindex Checker - Audisto Crawler	2.8 parsecs away Travel
\|	\|	Use our scalable indexability checker to test how a robots directive noindex, robots.txt directive, canonical link, hreflang or duplicate content affects the indexability and SEO of your website.	2.8 parsecs away Travel
\|	\|	seirdy.one Opting out of LLM indexing - Seirdy	3.0 parsecs away Travel
\|	\|	I added an entry to my robots.txt to block ChatGPT's crawler, but blocking crawling isn't the same as blocking indexing; it looks like Google chose to use the	3.0 parsecs away Travel
\|	\|	www.hanselman.com Choice amongst cross-platform .NET IDEs - VS Code, Visual Studio for Mac, JetBrains Rider - Scott Hanselman's Blog	21.1 parsecs away Travel
\|		A few years back, .NET development on a Mac was resigned to Mono and whatever ...	21.1 parsecs away Travel