robots.txt for Gemini formalised

[James Tomasino]

Just an FYI on the recent discussion around implied license for search engines and archival: These aren't rules baked into a spec, they're implications of the DMCA in the US and relevant case law, such as BLAKE A. FIELD vs GOOGLE (2016). The existence of a mechanism to disallow indexing was vital to that decision establishing implied license. Search engines, whether they be our lovely friend GUS or some future behemoth, can gather, index, and cache as they see fit because there is a mechanism for you to say no. That mechanism is the robots.txt and they have a strong case saying that the rules which govern it are already well established.

As much as I'd love to wave a magic wand and say, "it's all opt-in here" we don't really have any legal footing to do so.