It’s two years since we launched Mwmbl, the open source, non-profit search engine, on Boxing Day 2021. A good time to take stock of where we are and where we’re going. We’ve indexed over 100 million pages Thanks to our volunteers, who crawl the web using the Firefox extension and command line script, we’re crawling up to a million pages a day, as you can see on our stats page. There are around 50-60 users crawling on an average day.
  • Karna@lemmy.mlEnglish
    5·
    11 months ago

    Just a quick note - command line script : this can be run as a docker. If you have any home server (Raspberry Pi x) and willing to spare some server time, you can run as a docker container on it.

    • daoud@lemmy.mlOP
      4·
      11 months ago

      Thanks! Good to know that it runs on a Raspberry Pi - which one do you have? I remember someone saying it ran out of memory on their Pi.

      • Karna@lemmy.mlEnglish
        3·
        11 months ago

        I got a Raspberry Pi 4B (8GB). Though I installed mwmbl only recently, but havn’t noticed any memory leak as such; will keep monitoring though. BTW, i disabled logging for this container.

    • daoud@lemmy.mlOP
      2·
      11 months ago

      Good to know, thanks I don’t think it was that there was a leak, probably just had less RAM on their Pi