For the past several years Google has utilised several different crawlers to crawl content for smartphones and feature phones. These crawlers, that are specifically for mobile devices have been collectively called Googlebot-mobile.

Nevertheless, smartphones and feature phones have capabilities with substantial differences. There have been many instances when webmasters have inadvertently blocked smartphones from being crawled when they only intended to block feature phones. This issue prevented Google from indexing the content from some sites.

Google Has Created A New Smartphone Crawler

To add clarity to this situation and provide webmasters with improved control, Google will phase out Googlebot-Mobile. Then new smartphone crawler will be identified as “Googlebot”. However its user-agent string will still contain the word “mobile”.

This change only has an effect upon the smartphone crawler, Googlebot-Mobile.  The regular Googlebot crawler will not change and the two remaining Googlebot-Mobile crawlers will still reference feature phones in user-agent strings.

Indexing And Crawling

In order to test a specific site you can use a Webmaster tool called “Fetch as Google”.

An important point to note is that the new smartphone Googlebot will follow your robots met tag, HTTP header, and robots.txt directives rather than referring to Googlebot-Mobile.