Crawlers on Gemini and best practices

[Stephane Bortzmeyer]

On Fri, Dec 11, 2020 at 08:38:12PM +0000,
 colecmac at protonmail.com <colecmac at protonmail.com> wrote 
 a message of 29 lines which said:

> Gemini is based around the idea of radical familiarity. Creating a
> new robots standard breaks that, and makes things more
> complicated. There are existing complete robots.txt standards

The problem is precisely the final S. I can add that most of the
"standards" are poorly written and very incomplete.

> but let's just pick a standard that works and make it offical.

OK, that's fine with me.

> https://tools.ietf.org/html/draft-koster-rep-04

(Better to indicate the URL without the version number, to get the
latest version.)

It seems well-specified, although quite rich, so more difficult to
implement.