How to Block Spiders on the WordPress Install

Search engine "Spiders," also called "Web crawlers," are automated programs that search the Internet on behalf of search engines. Their primary purpose is to add new Web pages and updates to old pages to their search engine's index, so that people can find this information when they perform Web searches. Because most website owners want to disseminate their content to as many people as possible, spiders are normally welcomed by site owners. However, some webmasters may prefer to keep their sites private and less accessible to visitors. In WordPress, spiders are allowed access by default but can be blocked immediately after WordPress has been installed.

Things You'll Need

  • WordPress administrator status
  1. How To Block Spiders From the Entire Site

    • 1

      Log in to your WordPress admin area. This is located at

    • 2

      Click "Settings" on the large menu bar on the left of the screen, then click "Privacy."

    • 3

      Click the radio button next to the option labeled "I would like to block search engines, but allow normal visitors." Click "Save Changes" when you are done. Spiders will now be blocked from your entire WordPress site.

    How To Block Spiders From Specific Pages

    • 1

      Log in to your WordPress admin panel. Click "Plugins" on the left-side menu bar, then click "Add New." From this area you are able to install a great number of free plugins to your WordPress blog, offering a wider range of functions. One of these plugins can help you block spiders from crawling any posts you choose.

    • 2

      Type "WordPress Meta Robots" into the text field, then click the "Search Plugins" button. Find "WordPress Meta Robots" in the results list and click "Install Now," just beneath it. When the plugin has installed, click "Activate Plugin." (Note: "Robot" is yet another name for spiders.)

    • 3

      Click "Posts," then go to the post you want to block from Web crawlers. Click its title to bring up the editing screen. On the right side of the screen a new option has appeared called "Meta Robots." Click the drop-down menu, then click the "noindex, nofollow" option. Click "Update" near the top right of the screen to save your changes. Spiders are now blocked from this post. Repeat for any other posts you wish.

Tips & Warnings

  • Only WordPress administrators can install plugins -- if you are not a site admin, contact the site owner and ask them to either upgrade your permissions or complete this task themselves.


  • Photo Credit Hemera Technologies/ Images
View Blog Post

Geek Vs Geek: Windows 10: Should You Want Microsoft’s Next Contentious OS?