Crawlers on Gemini and best practices

[Stephane Bortzmeyer]

On Thu, Dec 10, 2020 at 03:07:50PM +0100,
 Côme Chilliet <come at chilliet.eu> wrote 
 a message of 4 lines which said:

> I don't see anything in the spec saying to stop at first match. I think you should read the whole response and apply all lines that matches your virtual user agent.
> So in this case for an archiver, all lines.

Then this example in <http://www.robotstxt.org/robotstxt.html> would
not work:

User-agent: Google
Disallow:
User-agent: *
Disallow: /

Because with your algorithm, Google would be disallowed (while the
comment in the page says "To allow a single robot[Google]").