robots.txt for Gemini formalised

[Sean Conner]

It was thus said that the Great Robert khuxkm Miles once stated:
> November 22, 2020 10:31 PM, "Sean Conner" <sean at conman.org> wrote:
> 
> > It was thus said that the Great Drew DeVault once stated:
> > 
> >> A web portal is a one-to-one mapping of a user request to a gemini
> >> request. It's not an automated process. It's a genuine user agent, an
> >> agent of a user. The level of traffic you'd receive from a web portal is
> >> similar to the amount of traffic you'd receive from any other user
> >> agent, and rate controls or access blocking don't make sense.
> >> 
> >> As the maintainer of such a web portal, I officially NACK any suggestion
> >> that it should obey robots.txt, and will not introduce such a feature.
> > 
> > What's the IP address of your web portal, so I can block it and prevent
> > the various webbots that will go through your web portal and index the
> > Gemini content without my consent?
> > 
> > -spc
> 
> I assume Drew's smart enough to block web bots from crawling his gemini
> portal. Just saying.
> 
> Just my two cents,

  Drew's proxy is a webserver in its own right:

	https://git.sr.ht/~sircmpwn/kineto/tree/master/main.go

  It checks for a GET request for "/favicon.ico" but not to "/robots.txt".
Every other GET request is immediately proxied to a gemini server.  I think
it was meant to run locally, but he made an instance available on the public
Internet.

  -spc