Crawlers on Gemini and best practices

[Stephane Bortzmeyer]

On Thu, Dec 10, 2020 at 03:20:18PM +0100,
 Petite Abeille <petite.abeille at gmail.com> wrote 
 a message of 8 lines which said:

> > On Dec 10, 2020, at 15:15, Stephane Bortzmeyer <stephane at sources.org> wrote:
> > 
> > Because with your algorithm, Google would be disallowed
> 
> An empty Disallow means allow all, no?

Yes, but the next Disallow add a restriction to everything.