robots.txt for Gemini formalised

[Robert "khuxkm" Miles]

November 22, 2020 10:31 PM, "Sean Conner" <sean at conman.org> wrote:

> It was thus said that the Great Drew DeVault once stated:
> 
>> A web portal is a one-to-one mapping of a user request to a gemini
>> request. It's not an automated process. It's a genuine user agent, an
>> agent of a user. The level of traffic you'd receive from a web portal is
>> similar to the amount of traffic you'd receive from any other user
>> agent, and rate controls or access blocking don't make sense.
>> 
>> As the maintainer of such a web portal, I officially NACK any suggestion
>> that it should obey robots.txt, and will not introduce such a feature.
> 
> What's the IP address of your web portal, so I can block it and prevent
> the various webbots that will go through your web portal and index the
> Gemini content without my consent?
> 
> -spc

I assume Drew's smart enough to block web bots from crawling his gemini portal. Just saying.

Just my two cents,
Robert "khuxkm" Miles