robots.txt for Gemini formalised

[Adnan Maolood]

On Sun Nov 22, 2020 at 7:30 PM EST, John Cowan wrote:
> Additionally: "Agent:" should specify a SHA-256 hash of the client cert
> used by particular crawlers rather than a random easy-to-forge name.
> Thus
> GUS should crawl using a cert and publicly post the hash of this cert.
> Then callers with that cert are necessarily GUS, since the cert itself
> is
> not published. (Of course it's still possible for a server to steal
> GUS's
> client cert.)

This doesn't seem very useful, as bad robots can simply ignore the rules
in robots.txt.