Microsoft blames Ubuntu update DNS issues for Azure services outage
Microsoft this week issued an alert noting that Domain Name System (DNS) issues are occurring for Azure customers who have upgraded Canonical Ubuntu on Azure Virtual Machines.
The issue specifically affected customers “running Ubuntu 18.04 (bionic) virtual machines [that were] recently upgraded to systemd version 237-3ubuntu10.54”, Microsoft’s alert said. Microsoft has also confirmed the issue in this Twitter post from Azure Support.
Canonical issued this announcement yesterday describing a “confirmed bug” for Ubuntu Systemd upgrades, where there are issues with DNS resolution. The announcement, however, did not mention the problems with Azure virtual machines.
Microsoft’s alert included a table showing that several Azure services were affected, such as Azure Container Apps, Azure Database for PostgreSQL Azure Kubernetes Service, and Azure VMware Solution. Presumably these services were only affected for Ubuntu upgrades using Systemd (see graphic snippet).
The problems began around 6:00 a.m. UTC on August 30, according to the alert, or around 11:00 p.m. Pacific Daylight Time on August 29, with a timeanddate.com account. The outages have apparently lasted over 12 hours so far.
At press time, Microsoft was investigating the issue, which has just been described as affecting Ubuntu 18.04 (bionics) upgrades using systemd. Here is Microsoft’s statement to this effect and its guidance:
This bug and a potential fix have been highlighted on the Canonical/Ubuntu site, which we encourage affected customers to read: https://bugs.launchpad.net/ubuntu/+source/systemd/+bug/1988119
An additional potential workaround that customers can consider is to restart the affected VM instances so that they receive a new DHCP lease and new DNS resolvers.
If you are running a virtual machine with an Ubuntu 18.04 image and are experiencing connectivity issues, we recommend evaluating the mitigation options above.
If you experience no impact to your Ubuntu 18.04 images, but have unattended security updates enabled, we recommend revisiting this setting until the Ubuntu issue is mitigated.
Former Microsoft employee Kevin Beaumont described DNS issues for Azure users in this twitter post. He noted the odd circumstance that this is an Azure issue which is blamed on Canonical.
“Azure blames Canonical (Ubuntu) but this only seems to impact Azure-hosted VMs,” Beaumont wrote.
The issue was also reported by Dr. Nestori Syynimaa, Senior Security Researcher at Secureworks, in this twitter post.
“These things happen all the time and will happen as long as people use technology,” Syynimaa wrote in that post, adding “not on this scale that often.”