mYfAVE'StOYOU: The Real Impact of the Google SmartPhone Crawler (Part 1): Situation Overview

Posted by Suzzicks

This post was originally in YouMoz, and was promoted to the main blog because it provides great value and interest to our community. The author's views are entirely his or her own and may not reflect the views of SEOmoz, Inc.

Not everyone realizes it, but Google has been serving different search results to mobile phones than desktop computers for a long time. Beyond that, they serve different results based solely on the handset you are using to search. The differences are often subtle, or focused on the order of Universal Results that are included in the mobile result-set, but Google is algorithmically trying to prioritize content that will work well on the phone that submitted the query, and give less priority to content that might not work on the phone. If you want to compare for yourself, MobileMoxie has a mobile search simulator tool that allows you see the results of one search query across three different phones at a time (cool!)

Now Google has launched a new smartphone crawler, and this will likely push the differences between desktop search and mobile search more into the mind of the average SEO. (Think: “Does Google consider this a smartphone or a feature phone or something else? I don’t know!”)

So let’s see if we can demystify the impact of this new smartphone crawler using some data. This is the beginning of a three-part blog post series, all focused on the new Google smartphone bot. This first post will focus on how it works and what sites will be effected. The next post will focus on how you can optimize your mobile content for the new smartphone bot best by creating effective mobile redirects. The final post will discuss common mis-indexing problems that mobile websites have and review how to protect your mobile content from search engine indexing problems.

Mongoose Metrics published a study that broke down how the top sites in the US are handling their mobile traffic. They divided the QuantCast Top Million sites (most US traffic’ d sites) into 3 categories based on how they were publishing mobile pages: Server-side redirection, JavaScript redirection and what they call ‘cloaking’ or selective serving of HTML assets. The study went deeper, and looked at results for iPhone, Android and RIM requests, but we can just look at the summary for all smartphones, which show the following stats:

For SEO, these results are each unique and relevant; here is why:

Server-Side Redirection

This is a mobile site-architecture strategy that uses two (or more) urls; one for the desktop or primary content, and one for the mobile content; designations can be added for tablets, WAP and other devices as well. The mobile content can be on a mobile subdomain, a mobile subdirectory or a totally different domain, and those decisions can all impact the content’s ability to rank. The mobile urls can be static and optimized urls or they can be temporary dynamic urls, which are usually stuffed with the exact paramaters of the mobile page request.

Until Google’s new smartphone crawler, this mobile SEO strategy relied on the rankings of the desktop pages which automatically redirect to mobile content when requested by a mobile phone OR building independent SEO value for the mobile pages so that they would rank on their own merit. This strategy sometimes also includes joining the mobile pages with their desktop counterparts by using the canonical tag to help share SEO value. What is most relevant for SEO is that that both versions of the page are left crawlable by search engines. This might be important if you target lots of WAP phone searches, which are still sometimes using a separate ‘mobile-only’ Google index and are not affected by the new smartphone crawler. This could also be important if there is a future algorithm shift that puts a stronger emphasis on mobile file size (which could still happen because it is so important for a good mobile user experience).

SUMMARY:

RESEARCH RESULTS FOR SERVER-SIDE REDIRECTION: 52.52% of the QuantCast Top Million Websites in the US may see an immediate user experience benefit from the new smartphone crawler. Once these sites have been crawled by the new smartphone crawler, they will be serving mobile content to mobile users from search results automatically, without their server having to process each of the redirect requests.

JavaScript Redirection

This mobile site-architecture strategy also involves a primary and a mobile url for each page and an on-page JavaScript redirect is included from the desktop page to its mobile counterpart. The strategy actually relies on the FAILURE of the search engines to crawl and execute JavaScript in order to work properly. Frequently this strategy relies 100% on the desktop page to rank well in smartphone search results, and the mobile pages are blocked from search engine indexing in the robots.txt file, to prevent the risk of duplication or confusion in the index. In some cases, this JavaScript is detecting specific phones, and redirecting to landing pages that are just built for those specific phones, which can get very involved, but is great for user experience. Unless the new smartphone crawler begins to execute JavaScript redirects (unlikely), sites that rely on this method will not benefit from Google’s new smartphone crawler.

SUMMARY:

RESEARCH RESULTS FOR JAVASCRIPT REDIRECTION: 2.15% of the top million websites are not going to benefit at all from the new smartphone crawler, but may have already had good results in mobile search rankings without indexing mobile-specific pages.

Cloaking or Dynamic Serving

This mobile site architecture strategy relies exclusively on one url that can display a page with different characteristics, depending on the device that requests the page. What content is served is determined by the server and something that is usually described as a ‘mobilization engine’ or a ‘transcoder.’ These are essentially databases of rules and content at various sizes or stages of degradation that can be sent, depending on the capabilities of the phone requesting the page. With this system, a desktop computer will get the full version of the site, but a mobile phone might be served a similar HTML skin, with smaller components switched in, to replace place of the larger elements that are served to the desktop computer, all on the same url.

A similar but less sophisticated version of this type of mobile publishing can be accomplished using mobile-specific style sheets and media queries to re-render or re-organize the content on the page based on the screen size of the device that it is requesting it (Responsive Design). This strategy uses only one url which might be appealing if you are trying to keep things simple for maintenance or SEO reasons. In both cases, bots will be served content based on the device or browser that they are emulating, so the smartphone crawler would likely (hopefully!) be served a smartphone-friendly version of the page unless the JavaScript is purposely made un-crawlable.

SUMMARY:

RESEARCH RESULT FOR DYNAMIC SERVING/CLOAKING: 45.33% of the QuantCast Top Million websites might be at a disadvantage in terms of bandwidth, user-experience and load-time. All of the dynamic processing is still required by their server to render the page each time there is a mobile page request. They will not likely benefit from Google’s new smartphone crawler, and might now be at a disadvantage, (in terms of the load time of the dynamically generated pages) where they probably had a slight advantage previously.

Mobile SEO is an ever-changing field. The search engines don’t all agree, and still don’t seem 100% sure how to best rank and evaluate mobile search results. Your best option is always to know how things work, keep a close eye on how your sites are ranking and do your own mobile testing wherever you can. Until the smartphone crawler many people were unaware that mobile search results were treated differently from phone to phone. Now you know that testing on your own phone might not be enough; you may have to address the new Google smartphone crawler with some well-planned mobile strategy. Hopefully you can use this analysis to help determine how your site will manage with mobile serving, and still please the new smartphone crawler.

Stay tuned for the next post in this three-post series about the new Google smartphone crawler. It will cover how you can optimize your mobile content for the new smartphone bot best by creating effective mobile redirects, then the final post in the series will review common mis-indexing problems on mobile and discuss how to update your mobile server settings to prevent search engine indexing problems.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don't have time to hunt down but want to read!