Hi,
At 13:23 GMT today the utopia-game.com website came under attack. This attack caused the component of the site that handles user accounts to suddenly consume all available CPU, which meant that there was insufficient CPU remaining to handle normal requests. As a consequence, many users experienced server timeouts.
We were monitoring server activity closely at the time of the attack, and became aware of the issue within minutes. We believe that the attack may have been deliberately timed to attempt to disrupt the scheduled resuming of game ticking at 14:00 GMT.
We were keen to limit the success of the attack, so spent the afternoon continuing to closely monitor the server, restarting it whenever the attackers managed to make it hang. A consequence of this is that for a few hours many players experienced intermittent issues accessing the server, suffering from occasional server timeouts and other errors.
We initially assumed that the behaviour was due to a severe bug that a single user was accidentally provoking. After putting in place some new code to track what requests were hanging the server, we realised that we were being intentionally attacked, either by a single person using an anonymizing network, or by a small coordinated group.
After a short game of cat and mouse, which involved us blocking IP addresses, and them/him/her coming back at us from another, we finally isolated the server code that was being exploited. We fixed the problem by updating a third-party component used by the code to a more recent version.
One unfortunate side effect of updating this component was that it temporarily broke a lot of forms on the site (between 17:00-18:00 GMT), because it introduced new security protection against another kind of attack (Cross-Site Request Forgeries). This required some additional configuration that took us a little while longer to finish.
As of this writing, everything is back to normal.
Thanks,
Brian & Sean
Reply With Quote