Crawlers on Gemini and best practices

[Stephane Bortzmeyer]

On Fri, Dec 11, 2020 at 09:20:25AM +0100,
 Stephane Bortzmeyer <stephane at sources.org> wrote 
 a message of 10 lines which said:

> > Like so?
> > 
> > gemini://example.com/robots.txt?user-agent=DetailsOfTheRobot ?
> 
> Good idea, this is what I do now.

Note that it is not guaranteed to work because of broken (IMHO)
servers. For instance,
<gemini://alexschroeder.ch/robots.txt?robot=true> redirects to
<gemini://alexschroeder.ch/page/robots.txt?robot=true> which returns a
code 50 :-(