[discussion] The matter of Robots.txt

[Alan Bunbury]

Why wouldn't we? We certainly have a lot of bots so it seems reasonable to have robots.txt.

I learned the value of robots.txt soon after setting up Remini, my Gemini proxy for Reddit. Many Reddit pages tend to link to a lot of other Reddit pages, so crawlers that visited Remini were sent down a rabbit hole which ultimately led to them trying to index all of Reddit (which is huge) via the proxy.

That's obviously not a usual case but I don't think it's *that* unusual either, in Geminispace. More generally, it seems obvious to me that there should be a (mostly) agreed-upon way to direct the behaviour of bots that visit one's capsule, so if there are good arguments against robots.txt I'd be interested in hearing them. I don't think this is strictly speaking a Gemini question though, as the robots exclusion standard is something quite separate to Gemini (or HTTP).

On 21/10/2021 13:41, Andrew Singleton wrote:
>
> I'm going to lead in with a question prompted by Sean's experiences.
>
> Do we even need a robots.txt?
>
> -- 
> -----
> http://singletona082.flounder.online
> gemini://singletona082.flounder.online
> My online presence