Sites hosted on SiteGround last week were not crawled by Googlebot, Google’s crawler. The problem appeared to have been a DNS issue between the provider’s partners (AWS) and Google according to the hosting company. The DNS issue was resolved after a few days, but with DNS it takes a long time to update and sites now start crawling again.
As I reported to Search Engine Land last week, Matt Tutt was one of the first to notice this issue – where it started on Monday, November 8th. SiteGround said it fixed the issue on Friday, November 12.
Here are the tweets from SiteGround:
We have reported the issue to Google and are working to resolve and identify the cause of the issue. We will keep you posted once we have more information or the issue is resolved.
– SiteGround (@SiteGround) November 10, 2021
Status Update: We are happy to inform you that we have implemented a fix for the Google bot crawling issue encountered by some sites. The websites are already crawled successfully. Please wait a few hours for the DNS changes to take effect. Thank you for your patience!
– SiteGround (@SiteGround) November 12, 2021
But was it completely fixed?
Why did I miss something? They just said they traced it back to that specific problem, no solution was found. I just tested one of my sites and got the answer below. pic.twitter.com/Mh42Eci1QC
– Matt Tutt (@ MattTutt1) November 12, 2021
There were more complaints after the 12th and then SiteGround said it applied additional fixes on the 13th:
We applied a patch last night and the sites are gradually being re-explored. Regarding CPU demand, please send us your domain by DM so that we can investigate and advise you further: https://t.co/zMYnoTzgn7
– SiteGround (@SiteGround) November 13, 2021
It seems that since the 13th SiteGround customers have been quieter about the issue, which may mean the issue is now resolved?
Google’s ongoing webmaster help thread, where Google’s Caio Barros helped deliver updates to those affected, also appears to have calmed down. It’s an interesting thread to browse, if you want to see it.
Google’s John Mueller also provided information on how Google is dealing with these DNS issues, in short – once the issue is resolved, things should get back to normal over time.
First of all, it happens from time to time, it can happen to us, it has happened to other big services (https://t.co/NjQeVR0Hrr). Short-lived outages sometimes go unnoticed because DNS can be cached.
– John 🧀 (@JohnMu) November 12, 2021
Googlebot usually performs DNS lookups on the same servers. So our new list of IP addresses can help you verify and authorize the list. You can test using the mobile friendly test tool and URL inspection in the search console. Https://t.co/32pLcCkiXz
– John 🧀 (@JohnMu) November 12, 2021
Because DNS server changes take a long time to propagate, unless you know they will persist for more than a day, it’s usually not worth the effort to move to another infrastructure. It is often easier to maintain DNS with hosting, so that any change of IP address can be updated automatically.
– John 🧀 (@JohnMu) November 12, 2021
There are usually no lasting effects from temporary outages like these. Technical issues come and go, we need to do our best to make sure that users can find their way to your wonderful sites through the search results.
– John 🧀 (@JohnMu) November 12, 2021
Due to the way Search Console aggregates URL information through indexing, the error count is unlikely to drop to zero immediately (although the errors are now gone) – that’s fine and just a problem. report.
– John 🧀 (@JohnMu) November 12, 2021
If you’re still around and wondering how to completely avoid this in the future: it’s difficult. Traditional hosting providers do an incredible job of keeping things running, and it’s extremely rare that I hear of outages.
– John 🧀 (@JohnMu) November 12, 2021
Send them chocolate (not cheese):
What would I do if a site I loved was affected? I would probably be angry and send chocolate to the people involved. They’ve done a great job, and I’m sure they’ll coordinate with the infrastructure folks to make it work even better. ??
– John 🧀 (@JohnMu) November 12, 2021
SiteGround is not a small hosting company, Wikipedia says the company hosts around 2 million domain names.
To be clear, I don’t think this was a big AWS issue, if it was, a lot more sites would have been affected. Seemed something specific to SiteGround and how they used AWS?
Discussion forum at Twitter and Google help for webmasters.
