I made a post here: https://lemmy.world/post/16949903

Hi. I raised an Issue on the crawler’s repo a couple of weeks ago. Still waiting for a response though …

Yeah, maybe. I’ll leave it with you …

Fixed now - a community called "wave" music caused a JSON deserialization error (because JSON uses " for its own purposes).

It’s been 2 weeks since I raised the Issue for 0.19.4 / 5 instances on lemmyverse - given that there’s been no response, we might have to give up with these lists anyway.

It depends. It seems like Lemmy batches up its activity to send to remote instances - so, per instance, it sleeps for a bit, then sends what it has. If both a Create and an Update are in the same batch, Lemmy just sends the Update. If they don’t happen to be in the same batch though, it sends both activities.

(this is outsider observation, not insider knowledge)

I wouldn’t do this personally, but if I did, I think I’d at least pipe the results to head -n 1 to only act on the first result.

‘subscribe to anything’ is handy, too. I’m subscribed to this post, for instance, so get notifications of new top-level comments.

For lemmyverse, the issue for it is here: https://github.com/tgxn/lemmy-explorer/issues/184

We’ll have to wait until that dev responds, or maybe until Lemmy devs change their mind about not providing a nodeinfo 2.0 response for 0.19.4 instances.

EDIT: ah, me and Blaze were commenting at the same time, it seems.

More and more stuff won’t feature as instances continue to upgrade to 0.19.4, as lemmyverse.net doesn’t yet support it (there was something like 582 instances listed yesterday on there, it’s 572 now).

The instance I’m on is defederated from links.hackliberty.org. I’m not certain why, but I looked at it to find it’s a place for conspiracy theories, including ‘9/11 Truth’ stuff, so it’s probably that.
There looks to be a bit of indecision about the right instance for a new ‘privacy’ community, so I’m just mentioning this in regard to that.

Nothing yet for today (Sunday), because the crawler at lemmyverse has failed. More info is here: https://github.com/tgxn/lemmy-explorer/actions/runs/9434433542/job/25986550190 - it’s bailed because there are 6011 communities missing from that run compared to the previous one. I’ve seen that number before, which suggests it’s a single instance with that many communities which is failing to give a response.

EDIT: the midday run worked though.

You’re right about the cause: Lemmy’s front-end isn’t giving its backend enough time to do everything it needs to do for an unfamiliar commentary.

It works better if you copy / paste the link into Search. MBin effectively redirects all these links to its Search anyway, so maybe that’s an option.

lemm.ee have upp’d to 0.19.4, and presumably turned on the Image Proxying thing - this is what I get for: https://lemm.ee/post/34040618

{"error":"unknown","message":"Request error: error sending request for url ([http://127.0.0.1:8080/image/original?proxy=https://ichef.bbci.co.uk/news/1024/branded_news/be3c/live/2a45fdb0-20ee-11ef-a3ab-e73c00cc3104.jpg)](http://127.0.0.1:8080/image/original?proxy=https://ichef.bbci.co.uk/news/1024/branded_news/be3c/live/2a45fdb0-20ee-11ef-a3ab-e73c00cc3104.jpg)): operation timed out"}

I think virtualizing the python environment (with venv) has become quite common, although my only evidence is that the server I’m using to send this was written in Python and uses one.

I’ve had that problem too (messing around with test instances) - for anyone else wondering why: it’s because the RSA keypair for me@mydomain has changed, and remote instances fail to validate message headers signed with the new private key, because they’re still using their copy of an old public key.

This isn’t really my area, but I’ll have a crack. From what I understand, Lemmy uses the ‘meta og:image’ tag to grab a thumbnail. Inspecting your site, I can see that that tag is in the html head. However, if you just ‘curl’ the URL, then it isn’t in the results. Using ‘curl’ for URLs from sites that are known to work in terms of generating thumbnails (theguardian and bbc), the tag is visible in the result.

This suggests that your site is using further scripting on page load to provide the meta tags, whereas perhaps Lemmy can only get them if they are provided immediately. There are other sites (like Reuters), who use additional scripting, that Lemmy is unable to get thumbnails for also (e.g. https://lemmy.world/post/16203031)

Most people will be unaffected ('cos their instance already has a copy), but trippingthroughtime has a broken outbox, so it won’t come with old posts if it’s a fresh discovery. Issue should be fixed in Lemmy’s 0.19.4 update.

I like to pretend that I’m !native plant gardening@mander.xyz when I don’t mow my lawn, but I’ve subscribed in the hope I’ll start doing it properly.

Accounts (which contain the private key that signs the headers in your posts, and the public key to verify) are required for ActivityPub to work.

Re: sorting posts not working - I don’t know. It looks like you’ve deleted the post you made about ‘sorting of posts not aligned’

Re: communities not updated - I found your GIF hard to follow, but there’s a straight-forward difference between the post list you’re seeing on lemm.ee and on programming.dev, in that the missing posts are all tagged ‘English’. (If you looked at lemm.ee when you’re logged out, you’d see the same list as on programming.dev).
I assume you’ve fixed it now, since this post is in English, but to recreate what lemmy-ui is doing:

#!/bin/bash

show_post=true
lang_id_undefined="0"
lang_id_english="37"

for page in {1..17}
do
  curl --silent "https://lemm.ee/api/v3/post/list?community_id=8024&page=$page&limit=50" |
  jq -rc '.posts[] | .post.language_id, .post.name' |
  while read line
  do
    if [ "$line" == "$lang_id_english" ]; then show_post=false; continue; fi
    if [ "$line" == "$lang_id_undefined" ]; then show_post=true; continue; fi
    if $show_post; then echo $line; fi
  done
  page=page+1
done