[tech] [psa] robustness (was Re: [ANN] A Gemini crawler, for statistics about the geminispace)

[Petite Abeille]

> On Dec 16, 2020, at 16:05, Stephane Bortzmeyer <stephane at sources.org> wrote:
> 
> I'm running a Gemini crawler

This is a very brave endeavor. Survivability is key in an experimental, buggy, hostile, or malicious environment. 

The same rules of cautiousness applies to any user-agent, but even more so to headless, automated bots.

There are many possible traps out there. Nothing new under the sun. The interweb has been through this for the last few decades: pranksters vs bots. 

As this is a known, solved problem, I will not bore you with specific details, assuming instead everyone knows what they are doing, eyes wide open.

Just be cautious. Assume hostility, aim for survivability.