Starting Off 2026 with R&D

News

Welcome to the first post of 2026! We’ve been focused on a few things internally that are related to Gensokyo Radio, Project Knightune, and other research which should benefit all web-related projects moving forward. It’s felt like a bunch of little things over the past month or two have come up, and much of what we worked on in January is in response to that. For more details, read on.

Towards the end of last year, we saw a few things happen that felt preventable.

First, we saw highly unusual podcast download activity from a small handful of clients where requests were an order of magnitude higher than normal. Since we utilize a budget-managed storage solution for hosting podcast episodes, this would exhaust the download budget and prevent any more downloads for a given day. We already utilize a cache system for our most frequently downloaded podcast episodes, but we’re looking into building a more robust system to prevent this from being an issue in the future.

Second, we saw some of our streaming endpoint servers’ certs expire. Normally these will auto-renew without any intervention, but the process changes a bit when working with cross-regional servers, and in our case this breaks the renewal automation we’ve been using for most of our websites. Fixing this means changing part of our process, and this has led to a larger dive into managing distributed systems which we’re continuing to work through.

Third, we’ve noticed that our regional endpoints will occasionally experience issues where they’ll stop accepting listeners. This can happen either by a software “segfault” where the software that accepts connections from listeners stops running, or by another issue where the software is still running but appears to be stalled. Both of these result in listeners being unable to connect to that specific endpoint, and while we can monitor the status of the software and the number of connections made to each endpoint in real-time, there’s still room for improvement in terms of ensuring that listeners can continue to connect even when an endpoint becomes unavailable.

Finally, as we continue to work through Knightune, we’re currently focusing on management and scaling systems. If you’re more familiar with terms like mitigation, high availability, failover, and so on, that’s the realm we’re working within at the moment. We’re putting time into learning how to create and manage scalable, robust systems, and in the system we hope to build, we also want to leave a path open for reducing or eliminating direct dependence on cloud-based infrastructure. Working on Knightune up to this point has largely been about careful planning, especially because it’s something we want to fund without external investment, and that’s important because we want that platform to be focused on helping artists, not investors.

That’s all for this one, stick around for more news related to the station and our other projects, and thanks for listening!

[Knowledge #215]

Suggested Posts