advertisement
Microsoft Reveals What Caused Global Microsoft 365 Email Outage
Microsoft has attributed last week’s global Microsoft 365 disruption to a software change introduced in July that interacted with increased network traffic and failed recovery systems, leaving businesses and government institutions unable to access email for much of the day.
In a preliminary post-incident report shared with customers, Microsoft said a change made to servers in its networking gateway caused internal processes to stall under certain workloads. The gateway handles connections to several Microsoft 365 services, including Exchange Online.
Although the change had been deployed gradually from July and operated without reported problems for more than three months, Microsoft said its testing had not exposed the flaw. The issue was more pronounced on newer servers with higher processor core counts, while its load tests had largely used lower-powered machines.
advertisement
The conditions that triggered the outage emerged on Thursday, 12 November, when Microsoft’s systems automatically redirected Teams traffic from a separate gateway to the one supporting Exchange Online. The affected gateway was already operating with reduced capacity following an unrelated problem the previous day. The additional traffic pushed the servers into a state where internal processes began waiting on one another, eventually exhausting processing capacity and memory.
Microsoft’s traffic management systems then compounded the disruption. Designed to redirect users away from unresponsive servers, the systems transferred traffic to healthy machines, which subsequently experienced the same problem. The failure spread across the gateway in less than two hours.
The company’s architecture is designed to isolate disruptions within separate regions and service units. However, Microsoft acknowledged that these safeguards did not contain the failure, citing another issue that remains under investigation.
advertisement
Recovery was also delayed because restarted servers were overwhelmed by waiting connections. Microsoft’s existing automated recovery processes were not designed to handle an outage of this scale. Engineers eventually began reversing the July software change, deploying the update across server groups and prioritising regions approaching business hours.
The disruption affected businesses across several sectors. Shipping company Maersk warned customers of delayed emails and updates, while insurer Aviva activated contingency arrangements. Airlines in India, including IndiGo, Akasa Air and Air India, also reported email support disruptions.
Other Microsoft 365 services, including Teams, OneDrive and SharePoint, recovered earlier because they could switch to alternative gateways. Exchange Online remained affected for longer because of the scale of its operations and its inability to use the same workaround.
advertisement
Microsoft says it has since introduced additional controls to monitor traffic redirection, limit how workloads are redistributed and increase capacity. It also extended email delivery windows to allow messages held in queues during the outage to reach recipients.
The incident raises questions about the resilience of organisations that rely on a single cloud provider for critical communications and business operations. While cloud platforms offer scale and operational efficiencies, outages can affect multiple business functions when underlying infrastructure is shared.
Microsoft has promised a final report detailing its corrective measures by 20 November. The preliminary findings have not yet explained why the safeguards designed to isolate failures across its network did not work as intended.