robots.txt for Gemini formalised

[Emilis]

On 11/23/20 2:30 AM, John Cowan wrote:
>
> By the same token, a public Gopher proxy (if there are any) should 
> respect "Disallow: gopherproxy".
>
> Other points:
> +1 for Allow:
> +1 for Virtual-Agent
> +1 for ignoring unknown lines
> Unsure what the difference is between Crawl-Delay: and Check:, but 
> having a retry delay is a Good Thing

A small nit-pick: if we use "Virtual-Agent" and "Crawl-Delay", we should 
at least use "gopher-proxy" instead of "gopherproxy".


--
Emilis Dambauskas
gemini://tilde.team/~emilis/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20201124/de97bb6b/attachment.htm>