Written by LunarSpotlight
Monday, 01-Aug-22 22:33:45 UTC
Written by LunarSpotlight
This week, we updated our Discord server’s entry process by re-enabling auto-verification via Hourai. This change will reduce friction for new server members by providing an easier path for server membership. For Discord users that are not auto-verified, MikoBot will still provide an automated application process for our mod team to screen members before entry.
We expect there to be more activity (and more for mods to do as a result), so we’d like to remind our Discord server members that “modping” is a pingable role in cases where a mod is needed. Use it wisely; unnecessary pings are likely to result in disciplinary action in the form of escalations issued by Hourai.
On Sunday, we quickly spotted and mitigated a web server load issue caused by a small handful of IP addresses located in Japan. The issue was caused by several continuous calls to our public JSON endpoint which is meant to function as a successor to the XML endpoint we initially created in 2012. This group of IPs has been seen making requests to both of these endpoints previously.
The frequency of requests is estimated to be approximately 6 million per day, effectively turning this behavior into a DDoS attack. Station service was impacted as a result, causing listeners to experience occasional audio dropouts during the activity. The attack is technically ongoing, but is actively being mitigated through Cloudflare. At the end of all this, we would like to remind folks who want to use our public endpoints (provided as-is) to be reasonable and use discretion.
In response to this activity, we also implemented a broad server-side rate limit. Essentially, this limits the number of requests that users can make in a given amount of time. Users may see images loading more gradually on the website as a result. This helps to spread server load out over a longer period of time to keep demand within what the server can handle. We’ll be fine-tuning this over the coming weeks to further automate mitigation, increase resource loading times to their previous allowance for regular users, and prevent our services from being impacted.
We also set up a new statistics monitoring suite which gives our team a much more granular look at server health, station health, current utilization, and usage trends. This will help us catch issues as they arise, and tell us more about where we should be focusing our efforts. This data has already helped us to confirm our choices for server regions in our multi-continental radio network.
Finally, we updated our listener logging system to use a new dataset from MaxMind which is regularly updated. This new dataset also has support for ipv6 addresses which are increasingly being used to connect to our station. This will give us a better picture of where listeners are from, as well as give a small boost in countries during our “Countries and Listeners” shoutouts during “Gensokyo Radio Live” broadcasts.
That’s all for this week, keep on tuning in to GR and thanks for listening!
One Year of Knowledge Posts (Discord) & Live #113
LunarSpotlight | Sunday, 20-Feb-22 17:39:00 UTC
Live #114 & Store Items Back in Stock
LunarSpotlight | Sunday, 20-Mar-22 19:30:46 UTC
Live #115 + Red & White Reimu T-Shirts Now Available
LunarSpotlight | Sunday, 17-Apr-22 19:23:26 UTC
Reviews, New Music, and Hoodie Preorder
LunarSpotlight | Sunday, 24-Apr-22 18:01:58 UTC
New Product Samples & GR Meetups on Google Podcasts + TuneIn
LunarSpotlight | Sunday, 13-Feb-22 20:16:00 UTC