Service degradation on Microsoft 365 (status.cloud.microsoft)

From the 10:11:39 PM UTC update:

Current status: We're continuing to pursue all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief. Our investigation has found that a core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding and contributes to the capacity loss.

To validate a potential recovery approach, we're fully isolating a portion of our service infrastructure, restarting the affected components, and gradually reintroducing traffic to confirm our services remain stable before applying this approach more broadly. We're also analyzing traffic patterns across our entire service infrastructure and evaluating whether isolating traffic within each region may allow affected regions to recover independently.

From the 9:41:46 PM UTC update:

Current status: We're continuing to work through restarting and validating the affected servers to safely return them to service, and we're exploring opportunities to expedite this process. In parallel, we're assessing whether reverting the small number of recent configuration updates we identified could help accelerate recovery.

From the 9:01:14 PM UTC update:

Current status: We've validated an alternative health check method on a subset of affected infrastructure, which allowed the servers in that subset to successfully return to service, and we're now applying this change more broadly. We're also preparing an automated recovery process to continually detect and restart unresponsive servers, to help keep them in service while this change progresses. Additionally, our review of recent changes has identified a small number of configuration updates that we're investigating as potential contributors to the issue.

From the 8:44 PM UTC update:

Current status: While validating the targeted change, we found that many of the affected servers have become unresponsive, which is preventing us from confirming whether the change is effective. We're restarting these servers so they can come back online with the change in place, and we're applying the same approach to additional affected portions of our infrastructure to validate the results more broadly.

Previous Status: While we haven't identified any recent changes that correlate with the start of impact, our investigation indicates the failed health checks may originate from a specific component on the affected servers that's responsible for processing them. We're deploying a targeted change to bypass this component on a subset of servers to validate whether this restores availability, and we've taken steps to prevent additional servers from being removed from service while we work to return the affected servers to service. We're also investigating why affected servers are experiencing elevated processor usage while handling requests.

I feel for those stuck in there. The outage going on for this long has got to be fucking brutal.

https://redmondmag.com/articles/2018/09/11/microsoft-explain...

> Current status: We’re continuing to pursue all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief. Our investigation has found that a core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding and contributes to the capacity loss.

Current status: While the servers we've returned to service initially remain healthy, we've observed that they can become unresponsive again once they begin receiving traffic. We're analyzing diagnostic data from these servers with our networking and platform experts to determine what's triggering this behavior.

Building on the protective traffic management measures we've implemented, we're also temporarily limiting which types of traffic can reach the affected infrastructure in specific locations. This will help us determine whether a particular type of request is contributing to the impact and allow us to restore service incrementally. Additionally, we're rebuilding a subset of affected servers using a previous software configuration to validate whether this prevents them from becoming unresponsive, and we'll expand this approach to additional servers if we observe positive results.

---

The 11:05:31 PM UTC update is is the first since the outage began that feels hopeful.

From the 11:06:31 PM UTC update:

Current status: We've begun returning a portion of the affected servers to service, and they're remaining stable so far. We've also shifted traffic and connections to alternative pathways wherever possible and have observed limited recovery in some regions.

We've streamlined our recovery process to expedite returning the remaining affected servers to service. To further support recovery, we're implementing protective traffic management measures in a specific location to reduce load on the affected infrastructure, and we'll expand these measures to additional locations if we observe positive results. As capacity is restored, users may experience a gradual recovery as their clients reconnect to the service.

---

The 10:11:39 PM UTC update is a little chilling. They don't know why what's happening is happening so they're doing experiments.

I do that with closed-source software when I don't want to bother trying reverse engineer it, but these guys have the code! Yikes.

I recognize that even w/ the code available there are likely behaviors that aren't likely to be anticipated because of the scale of traffic. Complex systems fail in complex and non-intuitive ways.

From the 10:11:39 PM UTC update:

Current status: We're continuing to pursue all available pathways to restore service, as our actions taken thus far have not provided the anticipated relief. Our investigation has found that a core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding and contributes to the capacity loss.

To validate a potential recovery approach, we're fully isolating a portion of our service infrastructure, restarting the affected components, and gradually reintroducing traffic to confirm our services remain stable before applying this approach more broadly. We're also analyzing traffic patterns across our entire service infrastructure and evaluating whether isolating traffic within each region may allow affected regions to recover independently.

I can conjecture some reasons. Datamining would be high on the list, since MS can read your mail and thus market to you more effectively. Governments might also be happier if MS can provide them with one-stop access (like Prism) to otherwise self-hosted e-mail accounts proxied through a central server.

I'm being an ass in saying this, because I'm aware human motivation is a beautiful kaleidoscope of variables, but I would have guessed at the venn diagram of 'those willing to compromise and use a Microsoft product' is mutually exclusive with that of the 'I'm hardcore enough to self-host email'.

I will say, however, that Outlook used to be one of the better desktop email programs, so I'll grant some legacy lenience.

Biggest data heist and most obvious US intelligence operation in human history. I can't fathom why AVs don't treat "Outlook New" as info stealer malware.

One of GitHub's services will be down again by next week.

This seems to be the smoking gnu, now what would it be ...

EDIT: Seems like now they confirmed that: "We've identified that affected servers are failing automated health checks, which is causing them to become unavailable, and this may be related to a recent change to a component involved in processing those checks. We're testing a configuration change on a subset of affected infrastructure to address this and, if successful, will apply it more broadly. In parallel, we've initiated a restart of all affected servers and are validating whether they're successfully returning to service."

First, you have every client retrying connections, whether automated or manual retries. This generates extra load over normal.

In addition, whichever systems come online first often get a ton of traffic sent to them, which can immediately bring them back down if you don't have ways to prevent it. You will get a wave of systems coming back online, a ton of traffic hitting them, and then falling over again.

Now, I am sure Microsoft is used to this and has procedures and systems in place to mitigate the issues (I know we did).

However, sometimes if the right combination of services are failing, even the strategies you had to avoid this sort of failure can themselves fail.

Or, rather, it's acting like it usually does.

If your org is using Intune or Defender EDR then you probably also are impacted.

And with Google is people complaining Chrome doesn't load their "pages". I bet they mean outlook.

Interesting: there's a huge correlation with everybody beginning to use LLMs.

And we dismiss it as being non-causal by saying it all started going downhill before the advent of LLMs?

I don't buy it. I think we're seeing the results of what happens when you hand the keys of the kingdom to LLMs.

Just clarifying this in general, when a company makes a voluntary offer to some segment of employees, especially at this size, it is a statistics game of how many people accept the buy out.

They did not make the offer to 25% of the employees to downsize by 25%, they did it to downsize by 5% (or something like that), and then made the offer based on how many they expect to accept. This is specifically covered in this line of the geekwire article:

> Microsoft didn’t provide a projection for how many employees will accept, but the company told GeekWire it anticipates a much smaller share than the roughly 30% who took its earlier voluntary retirement offer, which went to longtime employees more likely to take the deal.

They probably expect 10-20% of those offered to take it, which would result in a 2.5-5% total downsizing.