Assuming disallow-all, and some research on robots.txt in Geminispace (Was: Re: robots.txt for Gemini formalised)

[James Tomasino]

On 11/24/20 11:44 PM, John Cowan wrote:
> 2) Though it does not respect robots.txt, it is happy to make your
> content invisible to archive users by informal request (or, of course,
> by a DCMA takedown notice).

The Internet Archive does respect robots.txt, though they're not happy
about it and have written on the subject a few times. I included a
snippet in an earlier email with their user-agent.