Crawlers on Gemini and best practices

[Petite Abeille]

> On Dec 10, 2020, at 15:09, Stephane Bortzmeyer <stephane at sources.org> wrote:
> 
> Not exactly the same thing since my email was about order of
> User-Agent (when there is both "*" and "archiver") but, yes, Robot
> Exclusion Standard is a mess.

Right. Perhaps best to look at how things are actually implemented in the wild :)

Given your example -and a user agent of archiver- robots.txt Validator and Testing Tool at https://technicalseo.com/tools/robots-txt/ says Disallow.