Bad bots

[Kevin Sangeelee]

Tangentially, I recommend anyone writing crawlers first aggregate the
links, remove duplicates, then sort them randomly before processing as a
single batch.

For example:

cat found-links.txt | sort | uniq | sort -R >links-to-process.txt

This approach mitigates a lot of edge case problems with fully automated
crawlers, and ensures no one server gets inundated with requests.

Kevin

Sent from phone

On Wed, 9 Jun 2021, 09:35 Anna “CyberTailor”, <cyber@sysrq.in> wrote:

> Who owns a crawler running on IP 140.82.24.154?
>
> It doesn't respect robots.txt and makes requests too fast. I had to
> ban this IP with a firewall rule.
>
> This thread is open for complaints on other bad behaving bots.
>