Three technical failures triggered Microsoft’s global service outage

Three technical failures triggered Microsoft’s global service outage

Microsoft’s global outage stemmed from a software bug, a network load surge and reduced server capacity, disabling over 80% of gateway servers.

More than 80% of Microsoft’s network gateway servers stopped responding in less than two hours during last Friday’s global outage, according to the company. Rerouting traffic to machines that were still running overloaded those servers too.

Microsoft’s preliminary investigation, published on Monday and reported by The Guardian, attributes the disruption to three overlapping technical problems: a months-old software bug, an unexpected increase in network load and reduced server capacity. According to the report, neither a cyberattack nor AI agents caused the outage.

The worldwide disruption on Friday primarily affected Microsoft 365 services, including Outlook, Teams, Exchange Online and the Copilot AI assistant. Many users couldn’t access their mailboxes, and emails weren’t arriving. Calendars and certain files were also difficult to access. Some Teams functions weren’t working properly, though some users could still send messages and join online meetings without interruption.

The underlying bug came from a software change introduced in July for Microsoft 365’s network gateways. The modified software had run without problems for more than three months, and Microsoft’s testing had found no abnormalities. Under certain load conditions, however, the defect caused excessive CPU and memory use. Affected servers became overloaded, then stopped responding entirely.

Testing reportedly missed the problem because the hardware behaved differently in the laboratory than in production. On its own, though, the bug wouldn’t have caused a global outage.

Microsoft generally uses two separate network gateways for its services so one can take over if the other fails. This time, one of the gateways used by Teams wasn’t working properly, diverting its traffic to the other infrastructure.

That extra traffic arrived when the gateway servers were already short of capacity. An unrelated fault the previous day had taken more than 14% of their server capacity out of service. The increased load and reduced capacity together triggered the software defect that had been dormant for months. Requests piled up, and servers failed one after another.

Recovery took unusually long because restarting the servers didn’t fix the problem on its own. As machines came back online, the backlog of requests flooded them, and many stopped responding again. Microsoft eventually restored services gradually while continuously checking server health. Full recovery wasn’t completed until Saturday.

The consequences went beyond inaccessible inboxes. In some US prisons, unavailable Microsoft systems disrupted the processing of bail payments needed for release. Hospitals and healthcare facilities in Britain also reported disruption, with some locations having to return to paper documentation.

The Genius Geek
The Genius Geek

The Weekly Geek

Don't miss a thing: get our top stories in a weekly roundup.