Live #169 (Halloween), Staff Q&A, & Migration v2

News

Over the past couple of weeks, we held a Staff Q&A where we fielded questions from our community and more, broadcast our 169th episode of Gensokyo Radio Live on the 31st for a true Halloween show, and have begun transitioning websites and assets to a new system in an effort to resolve the stability issues we’ve seen since the migration earlier this year. For more details, read on!

On October 25th, we held a Staff Q&A in a similar style as our Meetup events in past years where we gathered in our Discord server’s stage channel, and if you’ve not had the opportunity to join either the Q&A or one of the Meetups, it’s like a panel at a convention. During the Q&A, we reviewed the last couple of years, dove into what we’re currently working on behind the scenes, and fielded questions from our community. We answered questions such as:

• Do you have a warehouse for your merchandise or is it on-demand?
• Are there any open staff or support roles?
• API and websocket questions related to community-created apps
• Thoughts on different kinds of pizza

Next, we broadcast Gensokyo Radio Live #169 on Halloween 2026 titled “A Halloween Show” because we’ve done themed shows in the past, but this time we turned it up with plenty of new music and excellent Halloween vibes. In fact, we had so much music prepared for this show that we ended the show officially after a couple hours, but played through the rest of the prepared playlist for our listeners to enjoy. This was a fun episode to prepare for, and hopefully our listeners enjoyed tuning in as well!

Finally, we’ve been fighting a digital battle for the past few months, and operationally it’s felt like a ball-and-chain has tethered me to my desk; while things have been running generally okay with occasional hiccups, when they do happen, those hiccups are both really annoying for website visitors and station listeners, and technically challenging to address. Within the past week, we’ve spun up a small cluster with a couple of new, temporary servers in order to get things migrated from the current system to the new system… er, the new-new system, since the current system is only a handful of months old at this point.

In diagnosing what the issues are with the current system, we’ve narrowed a handful of recurring issues to two main areas: networking and storage.

Networking

At the beginning of this year when we were testing things in our studio, we put together a system which had failover built in and was designed from the ground up to align with high-availability principles; in short, the system had multiple physical machines, and if one went down, the others were supposed to pick it up from there. Ideally, this results in websites and other systems that have little to no downtime. However, when we moved the setup into a datacenter for hosting (because hosting websites out of a house isn’t usually a good idea long-term), we realized that some of the networking techniques we were using in the test environment were not transferrable in the production environment.

In order to get the network side to work the way we intended, we needed to hack together an additional system that monitored things and told our provider when one of the servers were unavailable so that our websites could be pointed to a server that was still up. This had limited success, but ultimately we found that the system would run this switching operation unnecessarily, and it wouldn’t always result in our provider’s automations moving things over when needed. Eventually, we ended up reducing the public-facing network to use just one of the servers, and this stopped the unnecessary switching, but it also meant that our idealized “high availability” system was broken. This would require rethinking how to make this work.

Storage

Most of the websites and supporting systems that we run aren’t highly utilized with the one obvious exception being GensokyoRadio.net. On a server, the website and database for the radio station are backed by multiple pieces of software, each with their own requirements for resources like compute and storage. Compute resources have generally been plentiful enough, but storage has been a challenge, and not just for the obvious reason of having limited space to work with in production. In order to make it possible for a website to magically become available on a different server when the one it was on becomes unavailable, the files need to exist in more than one place; this provides redundancy and allows a website to effectively be rebuilt and rehosted quickly.

There are multiple ways to accomplish this and multiple pieces of software to make working with files easier. We chose a solution which met all of our requirements and functioned well in testing. However, we’ve seen multiple figurative cracks form in the way that this solution handles certain situations. Some situations are relatively easy, like turning things off and on again, but other times we’ve had to dig in for multiple hours, modify multiple configuration files, and restart multiple internal systems just to get storage back up and available to any of the websites we host (although it usually isn’t that bad). Usually when a website fails to come back up after some downtime, it’s related to a storage volume that the website depends on being unavailable due to space in the specific volume or something else still using the volume.

The space management gets especially frustrating and tedious to deal with because there are approximately 50 different volumes, each with their own bubble of space that they can use, and each one needs to have at least a little space available for the things that use them to keep working. Put this together with the high utilization that the website sees, and it’s common to see volumes fill up as well as server-level file pointer limitations that can cause pretty much everything to stop responding. Needless to say, a new solution that fundamentally works differently could do wonders to keep things running as expected.

Migration v2

With the main challenges of the previous system summed up, let’s look at what you can expect during this second attempt at migrating things to the new-new system:

• Migration will be relatively simple: v1 was difficult because it involved rewriting large parts of how things were configured, but those configurations need relatively simple modifications to work on the new-new system.
• Downtime should be relatively short: Right now, we effectively have two clusters online and public-facing; downtime should be limited to switching things like DNS and broadcast over.
• We’re already partially done: Station.moe and the forums are already migrated to the new system; the radio station and store have yet to be moved over as of the time of writing.
• The main system that ran network failover internally has been removed, and we’ll be approaching this differently in the new-new system.
• The storage layer has been moved to a system that should make storage management easier, uses a different way of handling files that is less likely to exhaust file pointer limitations, and handles file replication without requiring static provisioning, meaning it’ll feel like we have more space available overall on similar hardware.

That’s a lot for this update, but stay tuned for more news as we head into the last two months of the year, and thanks for listening!

[Knowledge #231]

Suggested Posts