News----good, bad, ugly? You decide (was Re: [spec] comments on the proposed gemini spec revisions)

[indieterminacy at libre.brussels]

Stephane,

Perhaps we can employ a stricter standard concerning poorly functioning bots?

For example, a federated 'naughty-step':
* If a server is being bothered by cretinous behaviour message a naughty-step site
* From time to time (participating) servers get a list of ips to (temporarily!) block
* The bots receive a (new) response code, pointing out that it is temporarily on a naugty-step
* The response code recommends a uri to better bot practices


Jonathan

Stephane Bortzmeyer <stephane@sources.org> writes:

> On Wed, Oct 20, 2021 at 09:19:49PM -0400,
>  Sean Conner <sean@conman.org> wrote 
>  a message of 33 lines which said:
>
>> > * disrespecting robots.txt, the protocol spec and common sense
>> 
>>   Yeah, tell me about it.  Here's the robots.txt I had (have):
>> 
>> User-agent: *
>> Disallow: /test/redirhell
>
> Speaking of specifications, remember that robots.txt is not
> standardized anywhere, even when you limit yourself to the Web (and it
> is worse for Gemini, which has no User-Agent: field). There are
> several documents floating on the net and the examination of the
> robots.txt of Gemini capsules indicate a lot of variety.
>
> (Your robots.txt is very simple and therefore fits into the common
> subset of all robots.txt documentation/software. But it is not the
> case for all robots.txt.)