[tech] [psa] robustness (was Re: [ANN] A Gemini crawler, for statistics about the geminispace)

[Peter Vernigorov]

I don’t think details are boring here. Would you mind listing some of the
problems and possible solutions/workarounds?

On Tue, Dec 22, 2020 at 12:57 Petite Abeille <petite.abeille at gmail.com>
wrote:

>
>
> > On Dec 16, 2020, at 16:05, Stephane Bortzmeyer <stephane at sources.org>
> wrote:
> >
> > I'm running a Gemini crawler
>
> This is a very brave endeavor. Survivability is key in an experimental,
> buggy, hostile, or malicious environment.
>
> The same rules of cautiousness applies to any user-agent, but even more so
> to headless, automated bots.
>
> There are many possible traps out there. Nothing new under the sun. The
> interweb has been through this for the last few decades: pranksters vs
> bots.
>
> As this is a known, solved problem, I will not bore you with specific
> details, assuming instead everyone knows what they are doing, eyes wide
> open.
>
> Just be cautious. Assume hostility, aim for survivability.
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.orbitalfox.eu/archives/gemini/attachments/20201222/505b3292/attachment.htm>