crawling, proxies, and robots.txt

colecmac at protonmail.com colecmac at protonmail.com
Fri Oct 23 19:12:45 BST 2020


Hello,

Both of these projects and ideas sound great! Maybe it's obvious, but just in
case: Make sure you respect the user agent of * as well, which signifies any
robot.


makeworld


More information about the Gemini mailing list