[SPEC] Encouraging HTTP Proxies to support Gemini hosts self-blacklisting

Sean Conner sean at conman.org
Mon Feb 22 01:32:27 GMT 2021
It was thus said that the Great Emma Humphries once stated:
> What's the use case behind blocking proxies? 

  Some people don't want web crawlers to crawl their Gemini space.  Web
crawlers can be ... stupidly bad [1].  For me, it's for web proxies that
*DON'T* have a robots.txt preventing web crawlers through the proxy.

  -spc

[1]	http://boston.conman.org/2019/07/09-12



Archive home | About this archive