Tuesday, August 06, 2024

Alvin's Not So Smart Web-Trap

Alvin starts to realise his smart Web-Trap (using robot.txt) to block webcrawlers etc, stops his website being indexed and thus no one can find him, but the AI scrapers can just ignore it.


So probably not such a great idea.

No comments: