Google apologizes, explains Gmail outage

Google has come out and apologized for the major two and a half hours of Gmail outage earlier this week. The problem started at 1:30 am PST, and though users in the United States are generally asleep, others in other time zones around the world were left unable to access their Gmail accounts.

According to the official Google blog, the outage apparently originated from a routine data center maintenance event at their European data centers. Normally, taking a data center down for maintenance is not a problem, as other data centers are designed to take over automatically.

According to site reliability manager Acacio Cruz however, "Unexpected side effects of some new code that tries to keep data geographically close to its owner caused another data center in Europe to become overloaded, and that caused cascading problems from one data center to another. It took us about an hour to get it all back under control."

It sounds to me like the buggy code resulted in a DDOS-style attack on Google's own data centers, as the same overwhelming pool of users are redirected from one data center to the next--and overloading them in turn. Much has been touted about the ability of Google's data centers to transparently shift their load from one data center to another--perhaps the company need to rethink and fine-tune that strategy.

In the meantime, the bugs have since been found and fixed. As compensation, Google will offer 15 days of free service to businesses and organizations that pay for the email service.

For more on this story:
- check out this article at PC Authority
- check out this article at Gmail Blog

Related Articles:
Google news from FierceCIO