Ajax Indexing Protocol

Webmasters should follow Google's steps to make Ajax searchable.

Ajax is a client-side Web programming technique that uses Javascript and XML or JSON (JavaScript Object Notation) to retrieve data in the background without affecting the specific Web page. Search engines have had difficulty crawling Ajax content to make it findable in search engines. In October 2009, Google set out an Ajax indexing proposal to make Ajax more easily crawlable by search engines.

Modify Ajax URL Fragments

Ajax uses the hash symbol in URLs, which Google's crawler traditionally ignores because hash symbols are also used for anchor tags on the HTML page. So to allow Google to recognize Ajax URLs, Google proposes adding a token in the form of an exclamation mark to the URL, so the URL:

http://ajaxexample.com/page?query#state

Would be:

http://ajaxexample.com/page?query#!state

Use a Headless Browser

Google suggests using a headless browser such as HtmlUnit in order to make Ajax more indexable. A headless browser doesn't have a graphical user interface and is used to provide content of a Web page to another application. The headless browser runs on the Web server and processes Ajax URL fragments, and then it creates static HTML code based on the client-side browser state. This HTML page allows Google to index the content.

Add "Escaped State" Query Tokens to the URLs

Google proposes that search engines access Ajax URL's through adding state information to the URL with a query token. According to Google, "The proposed URL would then become http://example.com/page?query&_escaped_fragment_=state." When the Google crawler comes upon the exclamation mark in the Ajax URL fragment, it will ask the server to replace the exclamation mark with "?_escaped_fragment_=".

Users See Original URL

Though the search engine crawlers would see a modified URL that would allow them to index the Ajax content, Google proposes for best user experience that they display the original Ajax page URL (for example http://www.originalajaxpage.com/page?query#!state) for users to see in the search engine results pages.

References

• Gravity Search Marketing: Examples of Sites Using Google's Ajax Indexing Specification
• Google Webmaster Central Blog: A Proposal for Making Ajax Crawlable
• Search Engine Land: Google Offers a Proposal to Make Ajax Crawlable

Resources

• HTMLUnit Browser