Salami

Asking the important questions

It’s a bare-metal server hosted by Hetzner, a German provider.

They just upgraded to a dedicated server for 180€/month today.

Unless I’m very mistaken you can not use the same account.

I bet you could do it if your instance didn’t pull in a lot of traffic.

If it did… I reckon that you might be able to pull it off to a certain extent so long as your internet package was good enough, but if you got hit with a Reddit-level flood of incoming users, your network almost certainly wouldn’t be able to keep up.

Even if it could, if you were consistently eating through all the upload bandwidth, I reckon you’d draw the eyes of your ISP and they might send you a letter kindly and respectfully telling you that if you don’t upgrade to a commercial line they’re not renewing your contract.

Best would be to have thousands of servers. I wouldn’t make them subdomains of lemmy.world, but people could create separate instances for large communities, for example. https://lemmy-selfhosted.com Scaling: if the Lemmy software gets tuned a bit better, I think the current server could host at least a million users. But let’s hope for more servers first…

Fwiw, he has been providing quite a lot of transparency in his posts to this community. He’s shared his hardware config in detail, posted maintenance posts with brief descriptions of what he’s doing, and replied to comments around specific config tweaks. I haven’t catalogued a list of links, but I’ve seen him do all of these things in the last 48h. It’s easy to imagine that all these things could be compiled in real time into a how-to, but it’s a pretty big deal just to keep the lights on right now, and pretty difficult to understand whether tweaks that helped your setup are generally applicable or only situationally useful and happen to perform well for your specific setup.

I’m sure we will see more high-performance Lemmy guides in the future, but at this point no one has more than 36h of experience with high-performance Lemmy. Give them a minute to catch up.

More like throw more $$$ at the problem has been the solution.

I definitely second the motion on it being a branding problem. Stuff like sh.itjust.works seem to me like something that dark basement tech nerds would come up with that is “edgy” and really only used by them and other people like them.

I’m not really into the ironic “edgy” aesthetic and part of the struggle with this transition for me has been orienting myself in the space because I don’t want to commit to some “sketchy” edgelord URL

I do think join-lemmy.org could possibly be changed to show server usage/capacity and uptime. When I initially signed up I went for lemmy.ml because what the heck is the difference? Honestly I was having all kinds of timeouts and thought the entire lemmy-verse was probably struggling. I was concerned that was the experience everyone was getting that they were going to leave because it is unsustainable.

But I ended up seeing a page showing the uptime of serversnand lemmy.world was 100% (at that point). So I figured I’d start an account here. HOLLY CRAP IT IS SO MUCH FASTER. I would have had a hard time sticking around if it all worked like lemmy.ml.

I started a community on lemmy.ml. Wish I would have done it here.

nah, I’m bit regretting not signing up on their instance. sh.itjust.works is a cool name and can be a brag point. lol. lemmy world is a bit too generalist, but I won’t migrate there as ruud (the admin of lemmyworld) is doing a good job managing the instance. I appreciate that. :)

I joined sh.itjust.works because of the name, but it seems to run pretty well.

For what it is worth, I looked at sh.itjust.works . Reason I choose beehaw.org was they were more local, and had more local content and users. Plus the server focus and values seemed to fit me better. Yes their domain is a bit odd but that was not a factor for me.

Yeah, I get it. Naming optics aside, it seems an instance with a lot of headroom relative to others, with a capable team. Would be near the top of my word-of-mouth options in spite of the idiosyncratic name.

Agreed. I have no idea what I’m doing, but lemmy.world sounded inviting - thus, I’m here.

It’s important to recall that last week the biggest lemmy server in the world ran on a 4-core VM. Anybody that says you can scale from this to reddit overnight with “horizontal scaling” is selling some snake oil. Scaling is hard work and there aren’t really any shortcuts. Lemmy is doing pretty well on the curve of how systems tend to handle major waves of adoption.

But that’s not your question, you asked if Lemmy can horizontally scale. The answer is yes, but in a limited/finite way. The production docker-compose file that many lemmy installs are based on has 5 components. From the inside out, they are:

• Postgres: The database, stores most of the data for the other components. Exposes a protocol to accept and return SQL queries and responses.
• Lemmy: The application server, exposes websockets and http protocols for lemmy clients… also talks to the db.
• Lemmy-ui: Talks to Lemmy over websockets (for now, they’re working to deprecate that soon) and does some fancy dynamic webpage construction.
• Nginx: Acts as a web proxy. Does https encryption, compression over the wire, could potentially do some static asset caching of images but I didn’t see that configured in my skim of the config.
• Pict-rs: Some kind of image-hosting server.

So… first off… there’s 5 layers there that talk to each other over the docker network. So you can definitely use 5 computers to run a lemmy instance. That’s a non-zero amount of horizontal scaling. Of those layers, I’m told that lemmy and lemmy-ui are stateles and you can run an arbitrary number of them today. There are ways of scaling nginx using round-robin DNS and other load-balancing mechanisms. So 3 out of the 5 layers scale horizontally.

Pict-rs does not. It can be backed by object storage like S3, and there are lots of object storage systems that scale horizontally. But pict-rs itself seems to still need to be a single instance. But still, that’s just one part of lemmy and you can throw it on a giant multicore box backed by scalable object storage. Should take you pretty far.

Which leaves postgres. Right now I believe everyone is running a single postgres instance and scaling it bigger, which is common. But postgres has ways to scale across boxes as well. It supports “read-replicas”, where the “main” postgres copies data to the replicas and they serve reads so the leader can focus on handling just the writes. Lemmy doesn’t support this kind of advanced request routing today, but Postgres is ready when it can. In the far future, there’s also sharding writes across multiple leaders, which is complex and has its downsides but can scale writes quite a lot.

All of which is to say… lemmy isn’t build on a purely distributed primitives that can each scale horizontally to arbitrary numbers of machines. But there is quite a lot of opportunity to scale out in the current architecture. Why don’t people do it more? Because buying a bigger box is 10x-100x easier until it stops being possible, and we haven’t hit that point yet.

Mind you, it can sound a lot like clown world which is a phrase Nazis and other groups against progress love to use.

Quit clowning around.

Ensuring there’s no data leakage in those cached calls can be tricky, especially if any api calls return anything sensitive (login tokens, authentication information, etc) but I can see caching all read-only endpoints that return the same data regardless of permissions for a second or two being helpful for the larger servers.

It’s also worth noting that postgres does its own query-level caching, quite aggressively too. I’ve worked in some places where we had to add a SELECT RANDOM() to a query to ensure it was pulling the latest data.

Agreed. Luckily, it’s only a temporary measure.

True, but that won’t make it any more or less stable. I suppose the fact that kbin can serve a page without Javascript would make caching common pages easier but you’ll still need to update the cache every time someone comments.

In theory it shouldn’t be that hard to take the kbin frontend code and use it as a template for Lemmy. There’s a good chance rewriting the UI from scratch is easier, but it’s all just HTML and CSS.

For super cheap, there’s Oracle’s free* servers. Restricted to about 50mbps in bandwidth but doesn’t cost you a penny. Get the ARM server if you can, those 4 cores and 24GB of RAM will serve you well.

For cheap, there are hosting providers like Contabo and Hetzner that will offer you servers for a fair price. You’ll need to do your own backups though!

OVH has some very cheap dedicated servers if you need more performance. Your uplink will be limited to 100mbps but you get dedicated CPU cores, dedicated hard drives/SSDs, dedicated RAM, everything. This comes with the risk that you’ll need to do your own backups because those dedicated components can die and there’s no failover (by default).

It’s hard to say what hardware you’ll actually need, but I’ve read somewhere that lemmy.ml used to run on a machine with 8 CPU cores to serve thousands of users before everyone migrated over. The admin of lemmy.world posts regular updates on how the server is doing, I think the 21k users and all of its federated communities are running on a 32 core CPU with 128GB of RAM. Based on this, 6 cores and 8GB of RAM should be more than enough to serve ±100 users with the responsiveness of the lemmy.world instance (probably better because you’ll have comparatively more bandwidth).

*=free forever but you do need to answer a call from a marketeer

Completely OT but it’s so nice to recognise usernames you’ve often seen around your “neighbourhood” on Reddit.

Looking at ruud’s post, he moved the instance to a pretty beefy server - it sounds like a large part of the stability is coming from overestimating performance requirements.