Crawlers on Gemini and best practices

Stephane Bortzmeyer stephane at sources.org
Sat Dec 12 16:57:32 GMT 2020


On Sat, Dec 12, 2020 at 05:52:39PM +0100,
 Petite Abeille <petite.abeille at gmail.com> wrote 
 a message of 17 lines which said:

> But to be expected.

I had to add blacklist support in my crawler to explicitely exclude
some... creative capsules.

> Sean mentioned using #fragment to the same effect.

That would be wrong since URI fragments are not sent to the server,
unlike queries.



More information about the Gemini mailing list