I am honestly kinda curious how exactly you manage your servers and keep them up to date, I feel like there is a linux patch every week or so, which would usually require a reboot. Do you all deploy live patching, how you become aware of critical stuff in your otherwise busy lives, RSS?
Removed by mod
How do you handle the database schema changes during updates? Have two databases and disable replication during the update? How do you sync changes that occur to the backup while/after the main is upgraded?
I think OP is asking about system updates. Zero time application deployment is a different thing.
Removed by mod
Removed by mod
Not much of an addition, but you’re absolutely right, in most systems that are expected to be highly available, there’s standard maintenance times, an agreement in place, and no critical use of the system is permitted to be scheduled in that regular time period. Any deployments are limited to that window, in case a rollback is necessary, data sync, etc.
All of that is in addition to the type of high availability stuff you’re describing.
From my experience database schema changes require for all connections to drop, but they tend to happen a lot less compared to other updates.
An LTS distro like Debian and Ubuntu doesn’t update too frequently. I’ve never tried livepatching. I install updates on the weekends and reboot only if necessary. Downtime is usually about a minute and my uptime monitors don’t usually catch it.
My lemmy.fan instance died. Something broke with federation and I’ve never been able to get it running again, even with a new database and a new subdomain. I gave up for now, at least until better error checking and recovery is implemented.
Perform automatic updates and reboot when necessary.
If one is serious about hosting this, it’s best to isolate the services. One container or VM or reach service, with (probably) physical hosts for the DBs.
Schema change is more involved, but backup then update. If you have read only db, it should sync the changes when reconnected.
Realistically, federated data will be re-sent if the recipient doesn’t respond, so a few minutes of downtime is not the end of the world. At least that’s how mastodon works - not sure about Lemmy but I’m presuming it operates in a similar fashion.
Couldn’t auto updates break things if they arnt tested?
They should be tested upstream of you, assuming you aren’t using customized (eg roll your own) versions of any of the ancillary software (php, pgsql, redis, etc). Generally configs are either merged or not adopted, and you can restrict version upgrades to non major releases, if there’s chances of breakages between them (eg moving from pgsql 9 to 10, etc).
Keep instance small, with all users in the same timezone. Use NixOS, let it update everynight automatically and safely. It’s good enough for a small service, downtime is mostly when people are sleeping.
