David Baszucki apologised for the long delay of restarting the service in a blog post, blaming the outage on numerous issues. Roblox users had a tough week on Thursday. According to Baszucki, a core system in the building to solve the problem quickly became overwhelmed, a result of a subtle bug while still affecting our server. He made sure to clarify that the problem was not caused by the peak of any external traffic or any other particular experience. Because of the increase in datacenter number, the failure primarily resulted in the failure. That meant that most servos at Roblox were not able to communicate and deploy effectively. Roblox made that difference after users speculated that the suspension was caused by an overabundance of users participating in a Chipotle promotion. Shortly before the start of the nightmarish holiday on Roblox, the restaurant chain started a holiday on Roblox with a giveaway of $1 million worth of free burritos. Even though it was difficult to diagnose the actual bug, recovery took longer than we would have liked, Baszucki wrote. After we figured out the root cause, we were able to solve this problem by combining performance a bit, performance tuning, re-configuration, and scaling down some load. As soon as the firm starts researching what happened, Baszucki said that Roblox is going to provide an outline of the condition. The company believes that there is no user in the world who has lost data. He wrote, Your Roblox experience should be back to normal, he wrote. Roblox has over 200 million active users across many platforms. Other available include iOS, Android, tablets, PCs, Xbox, Oculus Rift and Vive. The game is played for a maximum of 156 minutes per day by daily active users. Here is the full blog post from Baszuckis: As far as the local community knows, we recently experienced an extended outage across the platform. We’re sorry the length of time we took to restore our service. Roblox uses a key key, Respect the Community, and to make our community happy. The users started to notice difficulties with the use of the platform on Thursday afternoon October 28th. This became our highest priority immediately. Teams started working immediately to find the source of the problem and get the problem back to normal. This was an especially difficult outage because it was involved in a combination of several factors. A core of our infrastructure became overwhelmed. The reason for the delay is the presence of a subtle bug in our backend service communications. That wasn’t due to some emergence of traffic in the outside, rather than to special experience. This didn’t exist because we found out the number of servers in our datacenters increasing. That result is that most people at Roblox couldn’t communicate and deploy effectively. As long as we couldn’t diagnose the actual bug, we’d take longer to recover. If we successfully determined the causes of the problem, we fixed that problem by performing a leveling, re-configuration and scaling off some loads. By this afternoon, we fully restored our service. Until we complete our analyses, we will publish a postmortem with more details, as well as the actions to avoid such issues in the future. We, in addition, will implement a policy to build the power of our creators in an economically stable community through this outage. There are more details on this to come. In order to improve our respect the community value, we will continue to be transparent in our postmortem. So far as we know, there have been no lost data for player persistence and in this case your Roblox experience should be back to normal. If you have any problems using Roblox now or in the future, you can always contact our support team. We’re thankful for the patience and support by both our players and developers over the time.
