New Stream Uptime Monitoring System

News

This is more of a technically related article than anything else, but it interests everyone because it has to do with an important element of the stream: uptime.

Uptime is what fraction of a given period of time something is active or running normally, often times expressed as a percentage. In the case of Gensokyo Radio, uptime has been lower last month than it was the month before due to some intermittent software issues mostly out of our control.

There have been a couple of ways that we’ve attempted to cope with the server’s odd behavior up to now. First, we set up a monitoring system on stream data, and that told us if information (such as song title, artist, album, listeners, etc) was showing up where it should. When that data didn’t appear, it would send us (myself and Auritribe) a message letting us know that the server was down and intervention was required. Essentially, it was a basic notification system that couldn’t do anymore than that.

A couple days ago, I installed a more powerful tool which not only notifies us of the server’s status, but automatically intervenes and restarts the server. Figuring out how to do this has been more complicated than it should have been. A sysadmin or another otherwise technically inclined individual might say, run a cronjob. That was one of the first things we tried, but to make a long and boring story short, it didn’t work.

So, why is this a good thing? It’s quite simple: uptime will excel to near its maximum potential with the improved system. Previously, downtime would last anywhere from a few minutes to several hours, depending on how inopportune the timing was (obviously, if an admin is sleeping, they won’t get the email until after they wake up). With the new system, we’ve seen the stream recover, consistently, less than 20 seconds after the server stops responding.

I’ll tell you firsthand, this is a large weight off of my shoulders, and (hopefully) the end of a very annoying issue we’ve been dealing with for the past month.

Enjoy~

Suggested Posts