Crawlers on Gemini and best practices

[Stephane Bortzmeyer]

On Tue, Dec 08, 2020 at 09:47:57AM -0500,
 Natalie Pendragon <natpen at natpen.net> wrote 
 a message of 32 lines which said:

> Yes, you should respect robots.txt in my opinion. It's not compulsory,
> but it's currently the best way we have to respect servers' wishes and
> bandwidth constraints.

It is interesting to note that some robots.txt are quite broken (see
gemini://houston.coder.town/robots.txt)