Crawlers on Gemini and best practices

[Petite Abeille]

> On Dec 12, 2020, at 15:28, Stephane Bortzmeyer <stephane at sources.org> wrote:
> 
> Note that it is not guaranteed to work because of broken (IMHO)
> servers. For instance,
> <gemini://alexschroeder.ch/robots.txt?robot=true> redirects to
> <gemini://alexschroeder.ch/page/robots.txt?robot=true> which returns a
> code 50 :-(

Bummer. But to be expected. Specially from Alex who likes to, hmmm, experiment wildly :)

Still seems to be a reasonable approach. Sean mentioned using #fragment to the same effect.