Explore >> Select a destination
|
You are here 
|
ethanmarcotte.com | ||
| | | | |
www.jeremiak.com
|
|
| | | | | How I sniffed the user agent in an edge function to prevent some AI crawlers from accessing my site. | |
| | | | |
arstechnica.com
|
|
| | | | | Restrictions don't apply to current OpenAI models, but will affect future versions. | |
| | | | |
www.andrlik.org
|
|
| | | | | It is now clear that at least some AI companies are ignoring robots.txt that forbid them from scraping a site. Robb Knight wrote up a great guide for explicitly blocking those scraping bots via your Nginx config. However, this site is currently served by AWS CloudFront, which means that the content gets served without the request touching the source server. I was sure there had to be a way to do something similar with a CloudFront function, so I set out to try. | |
| | | | |
keepinguptodate.com
|
|
| | | This article covers creating a blog from scratch using the static site generator Eleventy (aka 11ty). Eleventy keeps things simple and as you'll see, enables you to very quickly create a fully functional site. | ||
