robots.txt for Gemini formalised

Drew DeVault sir at cmpwn.com
Sun Nov 22 23:02:18 GMT 2020


Feedback:

A web portal is a regular user agent, not a robot.

Maybe we could normalize robots fetching robots.txt with the query
string set to some useful identifiying information? This would allow
gemini administrators to make bot-specific rules, understand the
behavior of their logs, and get in touch with the operator if
necessary.


More information about the Gemini mailing list