Requests for robots.txt

[Sean Conner]

  I'm going through my Gemini logs, and I'm finding this:

remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua:1965/robots.txt" bytes=14 subject="" issuer=""
remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua/glv-1:1965/robots.txt" bytes=14 subject="" issuer=""
remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua/glv-1/handlers:1965/robots.txt" bytes=14 subject="" issuer=""
remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua/glv-1/handlers/filesystem.lua:1965/robots.txt" bytes=14 subject="" issuer=""
remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua/glv-1/handlers/sample.lua:1965/robots.txt" bytes=14 subject="" issuer=""
remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua/glv-1/handlers/userdir.lua:1965/robots.txt" bytes=14 subject="" issuer=""
remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua/glv-1/msg.lua:1965/robots.txt" bytes=14 subject="" issuer=""
remote=XXX.XXX.XXX.XXX status=51 request="gemini://gemini.conman.org/sourcecode/lua/glv-1/cgi.lua:1965/robots.txt" bytes=14 subject="" issuer=""

(I'm censoring the IP to protect the guilty here)

  I don't mind the crawling, but I am concerned about the references to
robots.txt.  In the web world, robots.txt lives at the top level and *only*
at the top level.  I don't think there's been a official response from
solderpunk about robots.txt, but I would expect it to be very similar to how
it works on the web---the top level only.

  But a clarification would be nice (either way).  In my opinion, it should
only live at the top level, but I can adapt to every "directory" as well.

  -spc