Microsoft blames Ubuntu update DNS issues for Azure services outage


Microsoft blames Ubuntu update DNS issues for Azure services outage

Microsoft this week issued an alert noting that Domain Name System (DNS) issues are occurring for Azure customers who have upgraded Canonical Ubuntu on Azure Virtual Machines.

The issue specifically affected customers “running Ubuntu 18.04 (bionic) virtual machines [that were] recently upgraded to systemd version 237-3ubuntu10.54”, Microsoft’s alert said. Microsoft has also confirmed the issue in this Twitter post from Azure Support.

Canonical issued this announcement yesterday describing a “confirmed bug” for Ubuntu Systemd upgrades, where there are issues with DNS resolution. The announcement, however, did not mention the problems with Azure virtual machines.

Microsoft’s alert included a table showing that several Azure services were affected, such as Azure Container Apps, Azure Database for PostgreSQL Azure Kubernetes Service, and Azure VMware Solution. Presumably these services were only affected for Ubuntu upgrades using Systemd (see graphic snippet).

[Click on image for larger view.] Figure 1. Azure services affected by suspected Ubuntu 18.04 (bionic) upgrade issue on August 30, 2022 at 06:00 UTC (source: taken from Microsoft’s Azure Status page).

The problems began around 6:00 a.m. UTC on August 30, according to the alert, or around 11:00 p.m. Pacific Daylight Time on August 29, with a account. The outages have apparently lasted over 12 hours so far.

At press time, Microsoft was investigating the issue, which has just been described as affecting Ubuntu 18.04 (bionics) upgrades using systemd. Here is Microsoft’s statement to this effect and its guidance:

This bug and a potential fix have been highlighted on the Canonical/Ubuntu site, which we encourage affected customers to read:
An additional potential workaround that customers can consider is to restart the affected VM instances so that they receive a new DHCP lease and new DNS resolvers.
If you are running a virtual machine with an Ubuntu 18.04 image and are experiencing connectivity issues, we recommend evaluating the mitigation options above.
If you experience no impact to your Ubuntu 18.04 images, but have unattended security updates enabled, we recommend revisiting this setting until the Ubuntu issue is mitigated.

Former Microsoft employee Kevin Beaumont described DNS issues for Azure users in this twitter post. He noted the odd circumstance that this is an Azure issue which is blamed on Canonical.

“Azure blames Canonical (Ubuntu) but this only seems to impact Azure-hosted VMs,” Beaumont wrote.

The issue was also reported by Dr. Nestori Syynimaa, Senior Security Researcher at Secureworks, in this twitter post.

“These things happen all the time and will happen as long as people use technology,” Syynimaa wrote in that post, adding “not on this scale that often.”

About the Author

Kurt Mackie is senior news producer for 1105 Media’s Converge360 group.

Previous Ubuntu 18.04 systemd security update breaks DNS in Azure • The Register
Next How DNS Security Can Jumpstart the Shadow IT Management Process