[tech] robots.txt format

[Sean Conner]

It was thus said that the Great René Wagner once stated:
> Hi,
> 
> simple question: 
> is the following robots.txt format valid in a form that the "disallow" is applied to all User-agents mentioned before?
> ---
> User-agent: researcher
> User-agent: indexer
> User-agent: archiver
> Disallow: about
> ---

  That will work, but you need to add a leading '/' to the Disallow line:

Disallow: /about

That will match any request starting with '/about', like '/about',
'/aboutthis', '/about/that', etc.  

> or do i need to be more chatty?
> ---
> User-agent: researcher
> Disallow: about
> User-agent: indexer
> Disallow: about
> User-agent: archiver
> Disallow: about
> ---

  That will work too (same thing about the Disallow: line though).  You can
read more about it at <http://www.robotstxt.org/>.

  -spc