robots.txt for Gemini formalised

[Solderpunk]

Hi folks,

There is now (finally!) an official reference on the use of robots.txt
files in Geminispace.  Please see:

gemini://gemini.circumlunar.space/docs/companion/robots.gmi

I attempted to take into account previous discussions on the mailing
list and the currently declared practices of various well-known Gemini
bots (broadly construed).

I don't consider this "companion spec" to necessarily be finalised at
this point, but I am primarily interested in hearing suggestions for
change from either authors of software which tries to respect robots.txt
who are having problems caused by the current specification, or from
server admins who are having bot problems who feel that the current
specification is not working for them.

The biggest gap that I can currently see is that there is no advice on
how often bots should re-query robots.txt to check for policy changes.
I could find no clear advice on this for the web, either.  I would be
happy to hear from people who've written software that uses robots.txt
with details on what their current practices are in this respect.

Cheers,
Solderpunk