robots.txt for Gemini formalised

Sean Conner sean at conman.org
Mon Nov 23 03:31:09 GMT 2020


It was thus said that the Great Drew DeVault once stated:
> A web portal is a one-to-one mapping of a user request to a gemini
> request. It's not an automated process. It's a genuine user agent, an
> agent of a user. The level of traffic you'd receive from a web portal is
> similar to the amount of traffic you'd receive from any other user
> agent, and rate controls or access blocking don't make sense.
> 
> As the maintainer of such a web portal, I officially NACK any suggestion
> that it should obey robots.txt, and will not introduce such a feature.

  What's the IP address of your web portal, so I can block it and prevent
the various webbots that will go through your web portal and index the
Gemini content without my consent?

  -spc





More information about the Gemini mailing list