How to Get Your Site Crawled Faster

A search engine spider must crawl your website before it can be indexed.

Many ways exist to get the word out about your new website. Methods include offline advertising, paying for search engine traffic with a pay-per-click plan or submitting articles to directories. However, most traffic-generating methods involve expense or time-consuming work. A free, hands-off process of gaining visitors is from organic search engine results. These are the listings that appear on a search engine such as Google, Yahoo or Bing when a user types in a keyword or phrase. For your site to appear within the listings, it must be relevant to the search term and included in the index of the search engine. For inclusion in the index, the site must first be crawled by search engine spiders.

Instructions

1. Submit your sitemap file to search engines. Using webmaster tools, such as Google Webmaster tools or Yahoo Site Explorer, lets the search engines know your new site exists online. In addition to helping you get your site crawled and indexed faster, webmaster tools provide you with other useful data, such as showing you which search queries are being used to find your site and sites that have a link pointing to your website.

2. Add your URL to search engines. Submitting your URL to search engines is another method of informing them that your site is live. The principal search engines no longer have a "Submit a site" link on the homepage, so go to any of the primary search engines such as Bing, Google or yahoo and type "submit a URL to XYZ." Replace XYZ with the name of the search engine to which you wish to submit your website. Open these results in your Web browser and submit your site to each search engine. Alternatively you can use a search engine submission service (see Resources).

3. Tweet your URL. Search engine spiders crawl Twitter, the micro-blogging platform, constantly for up-to-date, real-time news. Submitting a tweet containing your Web address and a snippet of news about your site could result in your website being spidered in as little as a few hours.

4. Submit your URL to multiple directories. Directories that contain lists of websites are regularly crawled by search engine spiders, and in turn, the URLs that are listed also get crawled. In many cases, acceptance of new listings is instant and you do not need to wait for a human to check your URL. To make directory submission simple, use directory submission software.

Tips & Warnings

• Placing a link on a well-established site or blog that is regularly crawled for fresh content will help your site get spidered quickly. If you don't have a well-established site, contact other established site owners and ask if you can submit a guest post that contains a link to your new site. Many site owners will make the exchange, as they will receive free content.

• Having your site crawled and indexed does not guarantee your site will show up high in search engine results. Concentrate on providing fresh, interesting content that people will link to from blogs and pass on to their friends for optimal results.

References

• Google Webmaster Central: Google Basics
• Yahoo! Help: Submitting Your Sitemap File to Search Engines
• Social Media Today: 10 Ways to Get a Website Indexed Fast; Lorna Li; 2011
• Google Webmaster Central: Google-friendly Sites