CDNs and Service Optimizations

News

Over the past couple of weeks, we wrapped up and deployed a few updates to the radio station’s website which has resulted in performance improvements related to serving website media and handling database inquiries. For more details, read on.

New CDN Implementation

The first improvement we’ll talk about is related to CDNs, or “Content Delivery Networks.” Since the beginning, all of the files that make up GensokyoRadio.net have been hosted together; it kept file management simple and we didn’t need anything fancier than just pointing to the website’s s folder to serve album art. Fast forward to the recent migration, and we quickly realized that stuffing a bunch of s into a repository is not really ideal; we saw this when we ran into issues running a simple file sync between the repo and the new deployment a couple months back. Repositories are good for managing files for websites and projects, and it’s normal to have a few supporting s as well for things like logos, icons, and maybe a few banners, but it’s not really meant to handle thousands of s for things like a library of album art. Couple this with the performance issues we were seeing in the new system, and it quickly became another item on our todo list.

To briefly cover what it does, a CDN is a network of servers that is typically geolocated so that there are servers physically closer to where users are. These servers store copies of content so that, when a website loads something such as an , that is loaded from the closest server in the network, and this often results in reduced load times and latency.

To implement a CDN for Gensokyo Radio’s website media (s and videos), we considered using our existing network of regional endpoint servers spread across three continents to handle caching. However, the original files still need to be hosted somewhere, and we opted to use a cloud hosting provider for this role. The number of hosted files is only about 2 GB which makes it inexpensive to choose basically any provider out there (but would take up about 10% of the space on our endpoint servers that only have a small amount of storage), and from there, we just needed to figure out how to wire up the provider to serve content requests. We decided to use Cloudflare for this since they’re commonly used for file caching, and for reasons I still can’t quite grasp, they provide this service for free. The last piece of the puzzle was updating the and media urls on the website to point to the CDN subdomain.

The CDN and updates to the website are already deployed in production, so any time you load an or video, it’s probably getting served through the network. The CDN has already handled over 220k requests over the past week with about 39% of media requests cached at this point. We’ve also been able to reduce the size of our repo by 99.4% since the s and videos are handled separately. This helps reduce server load on our own infrastructure, reduces load time for users, and simplifies project management.

Database Optimization

As cool as CDNs are, most of the improvement happened closer to home. Most page loads on Gensokyo Radio’s website involve calling extra information, whether that’s through the API for Now Playing info or by using a database query to retrieve information about specific circles and albums. New connections made to the radio station also involve the database because the system checks for things like bans, current listener info, and log data. The system also needs to store current listeners somewhere to enable song rating functionality since only active listeners are allowed to rate songs. Ideally, database queries are fast and optimized, and the process of serving a single webpage to a user operates in the realm of milliseconds. However, what we saw after the migration is that the website’s database would commonly be overrun with requests which, despite being handled well enough on the old system, were running significantly slower and would occasionally get backed up or even timeout on the new system.

It took a while to get to the point where we looked towards the database for performance improvements since we were tackling other issues related to network stability and internal service optimization earlier. Once we did look into the database, we spotted a few additional optimizations that helped take the database’s load average from around 1.0 to about 0.12. One of these optimizations was related to a call that occurred any time a user would connect to or disconnect from the radio station (so, a frequently used query). This had been previously optimized, but there was clearly still room for improvement (or perhaps settings didn’t carry over to the new system) as updating this reduced database load by about 80% on its own.

Next Steps

Since implementing the new optimizations, we’ve been able to scale down supporting systems and have reduced system RAM usage significantly. This will become more important as we transition the station’s broadcast system back into production in the near future. Once the broadcast system is back in place, listeners will start to hear radio IDs again and the previous sound settings will be implemented which will bring back a more comprehensive sound tuning system (see also: “The New Sound of Gensokyo Radio”).

After that, we’ve got mail, store, podcasts, and finally Station to return to in our project list, plus info about our next Live show shortly, so stay tuned for more and thanks for listening!

[Knowledge #228]

Suggested Posts