robots.txt for Gemini formalised

Natalie Pendragon natpen at natpen.net
Mon Nov 23 01:40:01 GMT 2020


This looks great! I'm excited to see this companion spec become more
formalized, and really like the categorical virtual agent design. One
thing that stuck out to me after a first read was the `webproxy` user
agent. What would you think of something like the following instead:

`proxy`
`proxy-web`
`proxy-gopher`

The prefixed design doesn't have any drawbacks as far as I can tell,
and would allow for more intuitively designed blocking/allowing
hierarchies. E.g., if there were 4 different types of proxies in use,
and you only wanted to allow one, you could be more restrictive with
`proxy` and less restrictive with the more precise, suffixed user
agent of the type you are okay with (e.g., `proxy-gopher`).

Emphasis on "more intuitively designed" - I realize you could
technically accomplish this in the current design by simply adding
`proxy` to the mix, but I think the prefix-based organization makes it
clearer and a bit more intuitive.

Warm regards,
Natalie


More information about the Gemini mailing list