robots.txt for Gemini formalised

[James Tomasino]

On 11/24/20 1:15 PM, A. E. Spencer-Reed wrote:
> On Tue, Nov 24, 2020 at 6:42 AM Nick Thomas <gemini at ur.gs> wrote:
> 
>> Thoughts? I appreciate this would impact on the ability of archivists
>> or researchers to capture geminispace, but I see that as a feature,
>> rather than an unfortunate side-effect :).
> I don't agree with archiving being disallowed by default. archive.org
> and others have saved me so many times, I can't imagine why one would
> not want an archive. If there is a reason I would much prefer an
> opt-out system for it.
> Why do you dislike archival?

Denying archival is already possible with robots.txt in its present form. We don't need to edit the spec for that either. If you want to avoid the internet archive you can use:

User-agent: ia_archiver
Disallow: /