I’ll start with this quote from Mike Rockwell’s blog:
I’ve been a happy Linode customer for years, but they experienced an outage Sunday morning that took my Cloudron server offline, impacting my Mastodon and Pixelfed instances. As of this writing they’re still offline and I’ve received an email letting me know that there is a potential for data loss.
I didn’t get a warning about data loss. This outage did wreck a sizable part of my Sunday, which was already stressful for unrelated reasons. And worse, it affected my customers’ weekends. I try to be patient with hosting providers because I’m one too, and I know how frustrating and unproductive it can be to feel piled on with complaints. But this outage was likely the most significant I’ve seen in the 10+ years I’ve been using Linode.
These events are a time to revisit past decisions. I was already feeling that I was overpaying for Linode. With a bunch of servers and databases, it’s a lot of money for a tiny company like Micro.blog, money that I could use to pay myself more or hire someone.
This year we’ve expanded to servers in Europe, hosted by Hetzner. I’m also now consolidating more of our S3-like storage to actually use AWS. I liked the redundancy of copying photos across multiple hosting providers. But S3 is reliable. If I had used it more, I would have avoided this outage.
From here the plan is to double-check that all photos have been copied to the right place. There will be housekeeping to fix some older links to photos, from 2020 and earlier, because of how I had to flip over the CDN yesterday during the outage. I should finish that work today and tomorrow.
Then we’ll continue using Hetzner more for our European folks, and maybe some in the US too, slimming down our use of Linode a little. I’d like our platform to be more balanced across providers.
