[users] [announce] geminispace.info - alternative search provider

[René Wagner]

Hi,

i didn't realize that Stephane wrote to the mailing list and 
answered him directly to his personal mail. So here we go again. ;)


It will work again - once the currently running indexing is done.
Unfortunately this is a shortcoming of the current GUS implementation.

Crawling and updating the search index are separate steps and the later one
locks the search index database.
Unfortunately, as geminispace grows, this becomes more of a pain than earlier 
cause indexing takes more time.

It seems that a mirror of a webpage with a huge archive popped up a few
days ago. I had to stop the crawl as it was still fetching this archive, i
probably need to exclude this mirror until we were able to improve the
performance of crawling/indexing.
Unfortunately i'm not that familiar with Python, so it may take some time.

Especially the data gathering parts (crawling/indexing) are currently bottlenecks. 
They are strictly sequential and single-threaded "one page at a time"
which will prolong these processes increasingly. 
But there are some more issues which arise as geminispace keeps growing,
GUS was not designed to index large capsulses which mirrors of webpages.

If you are interested in helping out with Python coding feel free to join,
every help is welcome:
https://src.clttr.info/rwa/geminispace.info/issues

I'm not sure if its feasible to improve GUS in a sustainable way or if we need
to start over and come up with a new design that honors the growth of geminispace 
(and is likely much more complex than GUS currently is).

regards
Ren?