Crawlers on Gemini and best practices

Stephane Bortzmeyer stephane at sources.org
Fri Dec 11 08:46:47 GMT 2020


On Thu, Dec 10, 2020 at 09:44:50PM +0000,
 Robert "khuxkm" Miles <khuxkm at tilde.team> wrote 
 a message of 24 lines which said:

> Not you. The idea is that you start with the most direct User-Agent
> that applies to you (in this case, `archiver`), and then if that
> doesn't say you can't access the file, go up a level (in this case,
> `*`),

Reasonable interpretation (more-specific to less-specific). Too bad
the "standard" is so vague.


More information about the Gemini mailing list