[users] [announce] geminispace.info - alternative search provider

[Natalie Pendragon]

I can't believe GUS has been around for over a year at this point.
There was one fairly substantial refactor about 4-5 months in, but the
architecture is still *mostly* the same as it was on day one. I think
that's kind of cool, because that means GUS has mostly survived a bit
over an order of magnitude of Geminispace growth - the first GUS crawl
indexed 26 domains! I'm not sure if you were looking for the story of
GUS, or an explanation of "why not Elasticsearch," but I'll tell the
story anyway in case it's interesting to anyone :)

GUS has two parts (similar to how Stephane broke down the problem in
another response): there's the crawler and there's the searcher. In
the beginning, the crawler wrote directly to a TF-IDF index as it
crawled. Then, the searcher, when run later, was simply a reader and
searcher of that same index.

On Sat, Feb 27, 2021 at 10:21:18AM +0100, C?me Chilliet wrote:
> I was kind of expecting to see a solution based on an existing
> search engine to emerge, such as elastic search, by implementing
> only the gemini specific parts, but I looked into quite a few
> project and all were terribly complicated?

It's interesting to note that, although I chose to build slightly
closer to the metal than, say, Elasticsearch, it's still basically the
same tech. Elasticsearch wraps niceties around Lucene, which is a
library for doing TF-IDF development. I chose to use a different
TF-IDF library directly in GUS - I thought for fun, I would try one
that was native to Python, to keep the project as self-contained as
possible. I found Whoosh [1], and while it's had a few sharp edges,
it's mostly been great to work with.

[1] https://whoosh.readthedocs.io/

Onto the big refactor I mentioned happened 4-5 months in. TF-IDF
indexes I would best describe as "fragile." You have to be careful
about interrupting any writes, you don't get easy transactions or
rollbacks like you do with most relational databases. This became
increasingly frustrating to have to deal with as part of the crawl,
which (again, as Stephane) is a difficult beast to manage in the real
world. Capsules get creative :) So I added one extra piece to GUS'
architecture - a SQLite database. The crawl writes to the SQLite
database, then the TF-IDF index is built afterwards, from what's in
the database.

I think GUS is about due for another rearchitecture of the crawler. A
lot of the recent hacking on GUS is being done by a few folks like
Rene with geminispace.info, so I would also be curious what their
ideas are, but I think there could be a lot of promise taking a
similar approach to what mozz did for his Gemini archiver [2].

[2] https://github.com/michael-lazar/mozz-archiver/

I also think the WARC output format is cool. It makes me think of a
world in which maybe we could even get rid of the need for crawling -
just provide tooling that folks can use to package their own capsule
into a WARC, then they submit it. It would be a more efficient use of
network bandwidth than crawling is, and it would also naturally be
opt-in, which after lots of reflection, I think would be a nice
property for a place like Gemini.

Well that's the story of GUS. Let me know if you have any questions or
want me to expound on any of this. I will close by saying I am very
excited and thankful to see new folks like Rene and Remco get so
involved with this project, and I am really looking forward to the
future of search in Gemini.

Warm regards,
Natalie