Crawlers on Gemini and best practices

[Sudipto Mallick]

> for example:
>
>     /a,/p all denied
>     /a/b,/p/q indexer,researcher allowed
>     /a/b/c researcher denied
>     /a/b/c heh allowed
>
now the researcher 'heh' may access /p/q/* and and /a/b/*
> and it may not access /a/b/{X} when {X} != 'c'
err. sorry. that should be: may not access /a/{X} when {X} != 'b' and
/p/{Y} when {Y} != 'q' (for all indexers and researchers, hmm.)
every one other that researchers and indexers may not access /a/* and /p/*

> other researchers may only access /p/q and /a/b/{Z} when {Z} != 'c' so
> they may not access /a/b/c
indexers may access /a/b/* and /p/q/*
ah.