I do the majority of my Lemmy use on my own personal instance, and I’ve noticed that some threads are missing comments, some large threads, even large quantities of them. Now, I’m not talking about comments not being present when you first subscribe/discover a community to your instance, in this case, I noticed it with a lemmy.world thread that popped up less than a day ago, very well after I subscribed.
At the time of writing, that thread has 361 comments. When I view the same thread on my instance, I can see 118, that’s a large swathe of missing content for just one thread. I can use the search feature to forcibly resolve a particular comment to my instance and reply to it, but that defeats a lot of the purpose behind having my own instance.
So has anyone else noticed something similar happening? I know my instance hasn’t gone down since I created it, so it couldn’t be that.
Does your server have enough power and workers to handle all the federated messages? Or is it constantly at 100% CPU?
The machine is a dedicated server with 6 cores, 12 threads, all of which are usually under 10% utilization. Load averages currently are 0.35 0.5 0.6. Maybe I need to add more workers? There should be plenty of raw power to handle it.
Yeah that sounds about enough to handle the load. How many workers do you use? And do you see any errors in your logs about handling messages? You could try to search for that particular thread to see if all replies are handled correctly?
Update: Did a
-fwatch of the logs for WARN messages while upping worker counts. Seems 1024 was the sweet spot. Upped further to 1500, and the warnings for expired headers have entirely stopped in large part. So it seems this was the solution.Thanks for your help!
I had it at the default 64, but I’ll try 512 and see if that helps. Nginx is configured to use 768, so I doubt there’s any bottleneck there. I did notice in the troubleshooting page there’s mention of searching the logs for “Activity queue stats,” but a grep of the docker log shows no results for that string.