Identifying robots (was Re: Open Source Proxy)
Sean Conner
sean at conman.org
Thu Jul 23 21:45:50 BST 2020
It was thus said that the Great Jason McBrayer once stated:
>
> This is cool, but when you stand it up, don't forget an appropriate
> robots.txt!
Question---HTTP has the Use-Agent: header to help identify webbots, but
Gemini doesn't have that. How do I instruct a Gemini bot with robots.txt,
when there's no way for a Gemini bot to identify itself?
-spc
More information about the Gemini
mailing list