/explore
|
You are here 
|
kevingimbel.de | ||
| | | | |
pxlnv.com
After Robb Knight found - and Wired confirmed - Perplexity summarizes websites which have followed its opt out instructions, I noticed a number of people making a similar claim: this is nothing but a big misunderstanding of the function of controls like robots.txt. A Hacker News comment thread contains several versions of these two arguments: [...] |
|
| | | | | ||
| | | | |
www.andrlik.org
It is now clear that at least some AI companies are ignoring robots.txt that forbid them from scraping a site. Robb Knight wrote up a great guide for explicitly blocking those scraping bots via your Nginx config. However, this site is currently served by AWS CloudFront, which means that the content gets served without the request touching the source server. I was sure there had to be a way to do something similar with a CloudFront function, so I set out to try. |
|
| | | | | ||
| | | | |
www.coryd.dev
AI companies are crawling the open web to, ostensibly, improve the quality of their models and products. This process is extractive and accrues the benefit to said companies, not the owners of sites both small and large. |
|
| | | | | ||
| | | | |
www.nbcnews.com
OpenAI said Tuesday it was unveiling GPT-4, a new version of its artificial intelligence software. |
|
| | | |||
