What are Search Engine Spiders?

Search engine spiders are software programs that sift through content on web pages and build lists of words that appear on those pages. This process is called web crawling. The program visits page after page, following each link and recording the content of each page as it goes along like a spider crawling through the pages. This content is then added to the search engine's index.

Different search engines use different approaches when sending out their search engine spiders. Some keep track of every word on a page. Others record meta tags, titles and subtitle words. Indexing the 100 most common words on a page is another tactic used by a search engine spider.

Links to websites are what feed search engine spiders. The more often that spiders see links to a website, the more often they visit. This gives the spiders more information to index a site and makes the site appear higher in the search engine's results for a search terms related to that site.

Search engine optimization (SEO) involves using specific tactics to draw search engine spiders to a site. Several ways to use SEO are to manually submit a site for inclusion in a search engine, cross link, use keyword phrases and use meta tags. This will help drive traffic and spiders to a site.

It also is possible to manually request a search engine spider visit. This involves submitting a site to a search engine for review and can be done for free or through paid submission, depending on the search engine. The search engine will then send their spider to the site. This might not help with page rank, but it will include the site in the engine’s listings.

Cross linking involves linking to pages within a website. This will drive search engine spiders to continue to go through the same pages of the same site, indexing as they go. Link spamming also is used, but it can harm a site page rank and SEO.

Using keyword phrases means including within the text of a website some phrases that a user would be likely to search for on a search engine. The search engine spiders see these, see how many times they appear on a page, and will index them. Eventually, if enough people are searching for these keywords, they will be driven to the site.

Meta tags are special codes placed within a web site that allow it to specify what the search engine spiders will index. It is important to remember that spiders will check the meta tags against the page content, and if they don’t coincide, the spider will reject them. Meta tags are not visible on a web site unless someone views the source code of the web site.