After Action Report: Station Intermittently Drops Listeners

News

On February 15th at 12:59am UTC, an issue was first reported by a listener related to station instability. Our initial investigation showed that approximately 6 hours prior to the first report, an issue began occurring on the station where all listener connections would intermittently be dropped by the server. This behavior appeared inconsistent and random, and we began looking at this issue from multiple angles including potential server misconfiguration, updates, DDoS, extraneous server network activity, insufficient CPU/RAM, and server media.

Some of the symptoms reported by listeners included:

• Radio endpoints sending 404 not found errors
• Mostly silent streams with music appearing in short durations
• Load errors when connecting via the web app
• Multiple songs displaying in info in quick sequence

Generally speaking, Gensokyo Radio uses two classes of servers for the radio station: a broadcast server which generates an audio signal for playback, and regional servers which listeners connect to. Regional servers simply relay the audio from broadcast and are not configured to generate any audio for fallback; this allows us to use low-powered machines while providing the benefit of being geographically closer to listeners.

Unfortunately, the current setup depends on broadcast being up and available; this is what we would call a “single point of failure.” When broadcast is unavailable, Icecast (the software listeners connect to on regional servers) automatically boots all connections and refuses new connections until there is audio to serve once again.

We found that “Liquidsoap” (funny name, but immensely useful), the software responsible for converting our playlist into the station’s audio signal, was intermittently disconnecting from broadcast’s version of Icecast; logs indicated the source signal was disconnecting internally from the rest of the system and would not reconnect until an internal timer, often behind by 20-40 seconds, was “caught up” to current. This meant there was no audio signal for regional servers to pick up, including the would-be fallback audio we use that listeners would recognize as audio themed after TOHO Clock, and listeners would be dropped.

While the exact cause of the internal disconnect remains unknown to us, we were able to find a couple similar reports by other Azuracast users who were on the rolling release, a version that is more frequently updated with newer features that haven’t yet been merged into the stable release. The rolling release recently moved to a newer version of Liquidsoap, and we speculate that this is where the issue stems from. More specifically, there is a line in the error log which appears to suggest an interaction is broken between Liquidsoap and the host OS, but this is beyond our area of expertise, so we’ll leave it to the maintainers from here.

Gensokyo Radio started using the rolling release version back in summer 2024 in order to begin using AutoCue, a system which introduced more advanced “radio-like” transitions between songs, ahead of the stable release.

Since our rolling release version was ahead of the most recent stable release, we were not able to simply revert to stable. Instead, we removed and reinstalled the broadcast server’s software, restored settings, and are currently in the process of uploading station media for regular playback. We will also need to manually restore podcasts, so expect a large portion of episodes to be unavailable until restoration is complete.

Thank you to those listeners who raised the issue and reported details to our staff via Discord, and to everyone else for your patience. Though we have many thousands of regular listeners around the world, it’s worth reminding folks every now and then that we’re just a small team of people who largely volunteer our time to keep this station running, and on a budget that has at times required us to opt for solutions that may not be the most optimal. We will take this experience, learn from it, and strive to serve our listeners the best we can with the resources we have.

As always, thanks for listening!

[Knowledge #195]

Suggested Posts