Identifying robots (was Re: Open Source Proxy)

[Solderpunk]

On Fri Jul 24, 2020 at 12:01 AM CEST, Natalie Pendragon wrote:

> There's been some talk of the generic sorts of user-agents in the
> past, which I think is a really nice idea. If `indexer` is a
> user-agent that both sites and crawlers had some sort of informal
> consensus on, then sites wouldn't need to worry about keeping up with
> any new indexers popping up.
>
> Some other generic user-agent ideas, iirc, were `archiver` and
> `proxy`.

I still really like this idea.  It will be a long and tedious
undertaking to build up some kind of rough consensus on a set of
user-agents with good coverage and granularity of different scenarios,
but I think it might be worth the effort.

It would be great to finally get a proper "robots.txt for Gemini" side
spec written up.

Cheers,
Solderpunk