☠Azure AD fell down last week, causing outages with Microsoft's Cloud properties Outlook 365, Office 365, the Azure Portal, and Teams were all affected.

​The root cause was a bug during key rotation, and I'll let the Azure Post Mortem team take it from here:

Azure AD utilizes keys to support the use of OpenID and other Identity standard protocols for cryptographic signing operations. As part of standard security hygiene, an automated system, on a time-based schedule, removes keys that are no longer in use. Over the last few weeks, a particular key was marked as “retain” for longer than normal to support a complex cross-cloud migration. This exposed a bug where the automation incorrectly ignored that “retain” state, leading it to remove that particular key.

Metadata about the signing keys is published by Azure AD to a global location in line with Internet Identity standard protocols. Once the public metadata was changed at 19:00 UTC on 15 March 2021, applications using these protocols with Azure AD began to pick up the new metadata and stopped trusting tokens/assertions signed with the key that was removed. At that point, end users were no longer able to access those applications.

Service telemetry identified the problem, and the engineering team was automatically engaged. At 19:35 UTC on 15 March 2021, we reverted deployment of the last backend infrastructure change that was in progress. Once the key removal operation was identified as the root cause, the key metadata was rolled back to its prior state at 21:05 UTC.

This is the second time in six months that Azure AD has gone down. This happened 6 months ago. These are growing pains for Microsoft's cloud endeavors, and the ops teams involved need #hugops. Microsoft being the "safe bet" for enterprises means in part being stable, and two enterprise outages in 6 months is a lot.

🤑Microsoft wants to pay you to build Cloud applications on Azure. I jest, but only a little. They want you to try out their new developer experience on Azure, and get your feedback on it.

🎁NuGet 5.9 is out and there's a nice blogpost by the nuget team on what's in it. Easier UI around version floating, a new "right click -> update", and some nice improvements in Visual Studio for NuGet.

🎁Microsoft releases a one click Microsoft Exchange mitigation tool. Download. Click. Mitigate the vulnerability.

📚Microsoft has Architecture guides for building .NET applications of all sorts. And of course, because Microsoft can't do anything without pushing Azure, the guides include how architect those applications in Azure. There's a reason why the Ebooks are free.

🏫Getting Started with the Microsoft Graph Toolkit is now free on Microsoft Learn. I had to google what Microsoft Graph was, and given the ... paucity of the Wikipedia article, I'm not sure anyone knows.

🖥Announcing Windows Community Toolkit 7.0 It includes a smattering of helpers for developing UWP apps, if that's your thing.

🎥#Include2021 is done, but you can view the videos if you register by March 24th. Include talked about diversity with voices from different industries.

🏰Microsoft Build is May 25th - May 27th, 2021. I don't have more info but when I do, you can guess where it'll be.