Microsoft has attributed last week’s global Microsoft 365 outage to a July update to its network gateway servers, saying the software bug remained dormant for more than three months before increased traffic and reduced capacity triggered the disruption.
According to the company’s preliminary post-incident report, the combination of a surge in network load and a decline in capacity activated the fault, resulting in the widespread service outage.
Microsoft published the report for customers in its Microsoft 365 admin centre on 15 November, The Register reported.
The company’s Executive Vice President for Copilot,
Agents and Platform,
Charles Lamanna,
shared an abridged version on X on Monday.
“A latent defect caused servers to become overwhelmed and unresponsive under certain load conditions,” Lamanna wrote.
The July update was specifically developed for Microsoft 365 network gateway servers and passed load testing before being deployed globally without any reported issues, Microsoft said.
However, the update contained a defect that triggered lock contention within an internal Windows process, eventually exhausting server CPU and memory resources.
Microsoft said its testing failed to detect the problem because load tests were conducted on less powerful hardware with fewer processor cores.
The report noted that the contention became more severe on newer servers with higher core counts.
The outage was triggered by two additional factors. Microsoft Teams failed over to a network gateway also used by Exchange Online, increasing traffic on the servers. Meanwhile, an unrelated issue from the previous day had already reduced the gateway’s available server capacity by nearly 14 per cent.
As affected servers stopped responding, network traffic was automatically redirected to the next available servers, which subsequently encountered the same software bug. Within two hours, more than 80 per cent of the servers were unresponsive at the outage’s peak, according to the report.
Microsoft said recovery was prolonged because traffic waiting to be processed overwhelmed servers almost immediately after they came back online.
The company added that its automated recovery system was not designed to handle a failure of such scale, making it difficult to restore services quickly.
“Since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change,” Lamanna wrote.
Microsoft engineers eventually developed an update that restored the Windows code path to its previous state. They deployed the fix to server groups in stages.
Teams, OneDrive, SharePoint and Office applications were able to switch away from the faulty gateways, allowing them to resume operations earlier.
Exchange Online, however, could not be moved off the affected gateways because of its scale, Microsoft said. This meant Outlook users experienced the longest service disruption.