Website Migration 2026 (After Action Report)

News

Over the past couple weeks, we have both started and completed a large migration project related to the website and its related services. This includes the website, database, websockets, and infrastructure. For more details, read on.

Migrating the website and its related services has been on our task list since this Spring, but with the phase 1 launch of Station.moe and the rebuild of Gensokyo.Store, the migration process kept getting pushed back. With the store in particular being down for essentially all of Q2, it became increasingly important to both a.) get the store operational again in order to continue funding the radio station and other projects, and b.) reduce operational costs since we’ve effectively been running two separate sets of infrastructure since earlier this year. In truth, both the initial deployment of Station.moe and the rebuild of Gensoyko.store were already deployed to the new infrastructure, and that helped to both reduce the number of sites running on the old system and to iron out the process of moving things to the new system.

Ideally, migration would have happened a couple weeks earlier, but the timing is tricky considering that most of our Live broadcasts take place in the second half of the month, so we ended up waiting until the end of July to do so. This matters because our server provider’s billing cycle runs on the 1st of the month. Because of this, the transition process was very noticeable and users were given information on how to listen to the radio station using the station’s endpoints while web-related services were unavailable.

In this migration, we chose to offline two servers with different roles: our web server (which also handled database, mail, and websockets), and our broadcast server. At this time, we’ve restored the website, database, and websocket deployments. Additional time was needed to work through several unforeseen challenges which would have been easy fixes on the old system, but became more complicated to deal with on the new system.

• For the webserver (supporting the website, web app, and API), we found that the large number of files and the size of the image asset library that supports browsing albums was unable to be brought over due to timeouts with the file sync method we were using. A separate process had to be developed and deployed in order to manually place files in their new locations without getting interrupted by timeouts. Adjusting timeout thresholds was ineffective for the number and volume of files that GensokyoRadio.net hosts. We may look into adjusting how static media is served on the website at a later time.

• For the database, we had tested a new system which allowed for automatic replication, backup, and increased service capacity, and that system is currently working for a couple of our other websites. However, with the large import process needed to bring Gensokyo Radio’s database over, we quickly found discrepancies with data across replicas which resulted in inconsistent query results. This meant that browsing music on the website would sometimes show everything as expected, and sometimes show nothing due to reading from a replica that never received the imported data. To solve this, we reverted to a known database system.

• For websockets (which drives the live info system), we used a very similar system as before, but the deployment of it was the most difficult part. There were several configuration settings that needed to line up both within and across the new system in order for websockets to work correctly, but once that was figured out, the system that serves live info has been running smoothly ever since.

There are still a couple significant systems that have not yet completed migration, namely the station’s broadcast system and the mail system.

• Broadcast is responsible for creating the radio station’s primary audio signal; this signal gets sent out to our regional servers where listeners can connect and tune in. Currently, broadcast is running on a temporary setup at our studio, and the goal is to redeploy this on the new infrastructure. This is probably the most difficult thing to migrate, and the process of doing so has effectively turned into its own side project.

• The other system left to migrate is our mailserver which remains unavailable for now. This will affect any messages sent by or received at our gensokyoradio.net or gensokyo.store addresses. Website account verification will be unavailable until the system is restored, and forum updates and summaries will be unavailable as well. Store order notifications should still be sent out by our payment processor. Station.moe emails should be unaffected since we use a different mailserver for this domain.

In summary, the most used, public-facing systems should all be restored at this time. We have noticed some lingering network issues that have cropped up a couple times since the migration completed, but the services themselves should be functioning normally at this time. Of course, if you notice any issues with any pages or systems within the website itself, please let a staff member know and we’ll get it sorted out. Our current focus will be on restoring the broadcast server so that this may once again be served at a datacenter.

That’s all for this update, stay tuned for more updates as we continue to get the remaining system deployments sorted, and thanks for listening!

[Knowledge #226]

Suggested Posts