/explore

Click through on any links that interest you or select the planets on the right to continue exploring the Outer Web.
You are here

 perishablepress.com
How to Block Bad Bots | Perishable Press
| | www.johnlarge.co.uk
Blocking aggressive Chinese crawlers/scrapers/bots - John Large .co.uk
3.6 parsecs away

Travel
| | Over the last few days Ive had a massive increase in traffic from Chinese data centres & ISPs. The traffic has been relentless & the CPU usage on my server kept spiking enough to cause a fault in my cPanel hosting. Im on a great hosting package with UKHOST4U and the server is fast & []
| | ethanmarcotte.com
Blockin' bots. - Ethan Marcotte
3.0 parsecs away

Travel
| | Here's how I'm blocking "artificial intelligence" bots, crawlers, and scrapers.
| | www.workingwith.me.uk
mod_rewrite, a beginner's guide (with examples)
4.7 parsecs away

Travel
| |
| | pxlnv.com
On Robots and Text - Pixel Envy
11.5 parsecs away

Travel
| After Robb Knight found - and Wired confirmed - Perplexity summarizes websites which have followed its opt out instructions, I noticed a number of people making a similar claim: this is nothing but a big misunderstanding of the function of controls like robots.txt. A Hacker News comment thread contains several versions of these two arguments: [...]