Last night, Lemmygrad went down due to a hardware glitch on our VPS provider. Unfortunately, they were unable to do anything to help me fix the issue, and asked that I reinstall from scratch 🙁 . They did inform me that the hardware issue at fault is fixed at least.
I do daily backups, so we lost a day of posts.
Pictures are in the process of being restored now, but it should take a day or so to restore them all.
Let me know if there are any other ongoing issues so I can fix them. Again, sorry everyone.
deleted by creator
Being involved in devops, I can relate to how much this sucks and I appreciate you taking the time to fix it. o7
Thanks comrade, ya these can be nightmare scenarios. I at least have several backup locations, so at most we lose a day’s worth of data.
I’m not involved in this type of work but I would just pull the plug, I can’t understand the amount of patience you need for this shit
Everything will be going fine for weeks, and then: oopsie poopsie everything’s gone!
Anecdotally, my instances have poofed into thin air even on AWS, so this seems not too rare. You use a managed database or something you’ve setup yourself?
Set up ourselves. Its postgres running in a docker container, with daily backups.
The only explanation is that someone posted cringe so big they went to mess with the VPS. Let’s act like it never happened but this better be the last time /s
Honestly how dare you 😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾😾🤖😾😾😾😾😾😾😾😾😾😾
Our VPS on what went wrong:
My cat
I mean, you can’t stop a kitty cat from having fun, don’t be a monster
Can relate.
we lost a day of posts
Spongebob_suffering.jpeg
Pain
Have this consolation prize:
more pain
No worries, these things happen. Look on the bright side, thankfully it was just one day loss and not more.
I’m glad it’s back, thank you for your work!
At least its back up now, glad you guys were able to fix it.
deleted by creator
nice to get an update, i was confused yesterday. every time i consistently can’t connect to lemmygrad i assume the worst hahaha
K, pics are done restoring.
thanks for your hardwork. I was concerned there was some type of ddos or something
Is it feasible to restore the posts and/or comments that were federated to lemmy.ml? I can see my stuff from yesterday there
(oh and also thank you for dealing with this so quickly)
I don’t think they can get sync’d correctly back unfortunately, because the original source is gone.
That’s unfortunate
might have to manually repost 👀
deleted by creator
Hrm… I’ll take a look.
Is there even 20 people on lemmy?
I was worried that the site got nuked for good because the outage lasted this long, very glad to know it’s not the case! I can imagine the horror you felt when this happened, data loss is a nightmare (I have to deal with this stuff at work). rip to the posts that got lost in the ether but you did what you could.
deleted by creator
I was worried, thanks
