[tech] robots.txt format

Stephane Bortzmeyer stephane at sources.org
Wed Jan 27 11:12:54 GMT 2021
On Wed, Jan 27, 2021 at 05:56:04AM -0500,
 Sean Conner <sean at conman.org> wrote 
 a message of 33 lines which said:

>   That will work too (same thing about the Disallow: line though).  You can
> read more about it at <http://www.robotstxt.org/>.

But do note that many Gemini capsules do not follow this specification
but one of the others (typically more complicated).


Archive home | About this archive