robots.txt for Gemini

[solderpunk]

On Tue, Mar 24, 2020 at 08:47:53AM -0400, Jason McBrayer wrote:
 
> Another type I can think of is "archive", for things that rehost
> existing gemini content elsewhere on gemini. Besides being another use
> case, this category also has the implication that it may make deleted
> content available (a la the Wayback Machine).

Yes, certainly!  This is actually quite an important use case.  Many
folks are unhappy that archive.org no longer respects robots.txt, while
on the other hand the archive.org folks argue that people were writing
robots.txt rules based on how they wanted search engine robots ot act
rather than considering archive bots.  Making it easy for Gemini server
admins to explicitly set different policies for the two kinds of bots
(if they want to!) seems a substantial improvement.

Cheers,
Solderpunk