Explore >> Select a destination
|
You are here 
|
kevingimbel.de | ||
| | | | |
www.coryd.dev
|
|
| | | | | AI companies are crawling the open web to, ostensibly, improve the quality of their models and products. This process is extractive and accrues the benefit to said companies, not the owners of sites both small and large. | |
| | | | |
pxlnv.com
|
|
| | | | | After Robb Knight found - and Wired confirmed - Perplexity summarizes websites which have followed its opt out instructions, I noticed a number of people making a similar claim: this is nothing but a big misunderstanding of the function of controls like robots.txt. A Hacker News comment thread contains several versions of these two arguments: [...] | |
| | | | |
tsak.dev
|
|
| | | | | With the recent news of OpenAI's web crawler respecting robots.txt and the ensuing scramble by seemingly everybody ensuring their robots.txt is blocking GPTBot, I was thinking if there wasn't a better solution to help our future AI overlords make sense of the world. As I am hosting all my sites on a tiny NUC using nginx and having previously played with its return directive I decided to reuse the same trick for visits of GPTBot. | |
| | | | |
mjtsai.com
|
|
| | | |||
