This new version introduced a system so that your instance stops sending out content to other instances that are supposedly dead / offline.

Unfortunately for some reason there’s false positives. When I checked comparing the results from a curl request vs the information in our Lemmy database I found over 350+ false positives.

In the DB there is a table called “instance” which has a column called “updated”. If the date on that column is older than 3 days, your server will stop sending any content to those instances.

For some reason I had entries that were dated as last being alive in July, while actually they were always up. If an entry is incorrect, you can fix it by manually using an update statement and adding today’s date. If your instance is not too large you can safely update all entries to today’s date and check if everything works as expected from then on any new content created on your instances.

The dead instances won’t have an impact unless your instance is larger and generates more content, thus it might be easier to simply update all entries and have Lemmy believe they’re all alive if you start noticing wonky behavior and don’t want to check one by one.

If you don’t know how to access the database run this command where domaincom is your instance domain without the dot.

  1. docker exec -it domaincom_postgres_1 busybox /bin/sh

  2. psql -U

(The default user is ‘lemmy’) You could technically do this is one single step, but it’s good to know the command to get shell access to the container itself if you didn’t know how to.

This should give you access to a postgres CLI interface. Use \c to connect, \dt to list tables and \d+ tablename to list table definition. You can also run SQL queries from there.

Try with this query: SELECT * from instance to list all instances and their updated date.

You can use other SQL queries to get better results or correct false positives. Just be careful with what you execute since there’s no undo.

  • Shadow@lemmy.caEnglish
    36·
    11 months ago

    One liner to get into postgres:

    docker exec -it lemmyca_postgres_1 psql -U lemmy

    Show only the instances not updated in past 3 days:

     select * from instance where updated < current_date-3;

    Fix those old rows:

    update instance set updated = current_date where updated < current_date - 3;
  • cwagner@lemmy.cwagner.meEnglish
    191·
    11 months ago

    Ohh, that is why every few days, for a few hours I stop being able to post to Lemmy.world, aka the centralized lemmy instance.

  • MarsAgainstVenus@fedimav.winEnglish
    11·
    11 months ago

    If you used Lemmy-Easy-Deploy, the docker name is lemmy-easy-deploy-postgres-1

    docker exec -it lemmy-easy-deploy-postgres-1 busybox /bin/sh

  • Nix@merv.newsEnglish
    5·
    11 months ago

    Very useful thanks for sharing! We should have an admin tips community to collect information like this

  • 1984@lemmy.todayEnglish
    51·
    11 months ago

    Thank you but is this really safe?

    Just asking since there was db errors that messed up other people’s instances, or so it seemed.

    • Wander@yiffit.netOPEnglish
      5·
      11 months ago

      It should be safe as long as you put in a valid timestamp and not some other value. If you run a large instance, then you run the risk of pseudo-DDoSing yourself by sending a large amount of requests to dead servers, but unless you’re a large instance you shouldn’t have to worry about that.

  • Xanvial@lemmy.worldEnglish
    3·
    11 months ago

    Is it possible to add some scheduled trigger to check all instances in that table regardless of the last updated value? Shouldn’t too frequent but enough to avoid fully defederating some instances

    • Wander@yiffit.netOPEnglish
      4·
      11 months ago

      In theory that’s what Lemmy now does every day, but I have no idea why it fails to update some instances sometimes. Instances which are very much alive at 12AM which is when this gets executed.

      • redcalcium@lemmy.instituteEnglish
        9·
        11 months ago

        If you’re unlucky enough to have lemmy.world momentarily down at the exact same moment your instance doing the check, then it’s possible. Also, lemmy.world implements rate limiting via cloudflare, and their admins probably not whitelisting requests from the dead instance checker.

        I think instance owners that use cloudflare should whitelist /nodeinfo/2.0.json so automated requests to that endpoint is not blocked by cloudflare’s bot protection (which can lead to failed daily instance check on instances you federated with).

        • Wander@yiffit.netOPEnglish
          4·
          11 months ago

          Good point. I think that might be it actually. This could be the reason.

      • Terrasque@infosec.pubEnglish
        2·
        11 months ago

        I hope that’s not 12am on the dot for all instances, with no fudging of the check time.