Last week Eric Enge and Mark Traphagen of Stone Temple Consulting hosted a Virtual keynote with Google’s Gary Illyes
For those who don’t know the name, Gary is a Webmaster Trends Analyst at Google Switzerland GmbH and a frequent speaker at search conferences and webinars. There’s a lot of good info in this session and I highly recommend watching the whole thing if you’re so inclined, but while the cool kids are discussing advanced stuff like Rank Brain, I’m going to show you how to fix something that may be holding back your site, right now: Mobile search issues caused by blocked resources.
Discussion begins at 15:08:
EE: “What are the most common problems people have on their mobile sites?”
GI: “I would say that robotted resources are still the biggest problem.”
What do files which control the site’s appearance have to do with Google’s search index?
In order to determine if a page is mobile friendly or not, Google attempts to render the page, bringing in any necessary resources like javascript, css and image files. If Google cannot access something because the resource is blocked by robots.txt (or some other method), they might not get the correct content for that page or be able to accurately assess how it displays across different devices.
For those concerned about making these files available, Gary pointed out that “we’re not going to index those resources anyway, so there’s no good reason not to allow us.”
Background
This is not a new issue. It first appeared as an update to Google’s Technical Webmaster Guidelines in October 2014, accompanied by an announcement on their webmaster central blog on 10/27/2014, which said in part:
Disallowing crawling of Javascript or CSS files in your site’s robots.txt directly harms how well our algorithms render and index your content and can result in suboptimal rankings.
In case anyone doubted that they were serious, Google began sending out blocked resource warnings to webmasters via Google Search Console in late July, 2015.
Is your site affected?
So how can you tell if this issue affects your site?
Google Search Console – Blocked Resources Report
This shows you what resources are blocked throughout your site and the pages affected.
That said, the data in Google Search Console isn’t always up-to-date, so you might want to sanity-check a few pages via another method, especially if you’ve made changes recently.
Also in Google Search Console, you can test your pages individually with Fetch and Render as Google, which will show you real-time results.
What if your site’s not verified in Google Search Console?
I highly recommend that you add it for a number of reasons. But meanwhile, you can use Google’s standalone Mobile Friendly Test . This reports on a single URL, not the entire site. The Mobile Friendly Test also shows how your page would appear on a mobile device. It’s possible for a page to be deemed “mobile friendly” yet have incomplete content due to blocked resources, leaving Google unable to index it as intended.
There’s a nifty little tool by Adam Whittles and Kostas Voudouris at urlitor.com which submits bulk URL’s to the Mobile Friendly Test. The results show the number of blocked resources on each page. Unfortunately it doesn’t show WHICH resources are blocked, but you can quickly pre-screen multiple pages this way and check back with the Mobile Friendly Test to get the details.
The solution
Assuming you’re using your robots.txt file to block access, here are the edits you need to make.
Are you explicitly blocking just the needed resources?
At the risk of stating the obvious, if you have robots.txt directives blocking individual files or folders containing only these resources, just edit your robots.txt file and unblock them!
Are the resources in another directory along with files you still need to block?
You have a couple of options. You can explicitly allow Googlebot to access the files …
Or better yet, allow access by filetype. This will save you a lot of headaches in the event you add more files or change the filenames. (Note that I added image filetypes, since images are also referenced in the Webmaster Central Blog announcement.)
Are the resources hosted on a 3rd party site?
If the resources are for something on another site which is only displayed on your site (for example, an ad banner), it shouldn’t be an issue. However, if it affects the layout or appearance of your site (like a javascript framework hosted elsewhere, a CMS or an ecommerce platform) or if it generates indexable content, that is more likely to be a problem – and one to which there may not be a satisfactory solution.
In the keynote, Gary mentioned that Google is working with various CMS’s to unblock these resources for users of their respective platforms. Meanwhile, opening a ticket with the support team for the product in question is probably your best bet.
Whatever you do, always test your robots.txt file in Google Search Console afterwards, to make sure the changes you made are working as expected!
Leave a Reply