Requests for robots.txt

[Natalie Pendragon]

FWIW I'm 99% sure those are requests from GUS, and I agree that it
should be top level only. That was a regression in GUS' crawling code,
which I've now fixed!

I'm still very happy to accommodate more official guidance on how
robots.txt should work, but in the meantime (and in the absence of any
more regressions, eep!) I plan to check top-level-only robots.txt.

So sorry about the :bug:!