[tech] robots.txt format
René Wagner
rwagner at rw-net.de
Wed Jan 27 09:40:39 GMT 2021
Hi,
simple question:
is the following robots.txt format valid in a form that the "disallow" is applied to all User-agents mentioned before?
---
User-agent: researcher
User-agent: indexer
User-agent: archiver
Disallow: about
---
or do i need to be more chatty?
---
User-agent: researcher
Disallow: about
User-agent: indexer
Disallow: about
User-agent: archiver
Disallow: about
---
kind regards
René
More information about the Gemini
mailing list