How do Search Engines Work?

Brendan McGuigan

Search engines are basically computer algorithms which help users find the specific information they’re looking for. Different ones work in different specific ways, but they all utilize the same basic principles.

Internet search engines are designed to help users find what they seek online.
Internet search engines are designed to help users find what they seek online.

The first thing search engines have to do in order to function is to make a local database of, basically, the Internet. Early versions just indexed keywords and titles of pages, but contemporary ones index all of the text on every page, as well as a great deal of other data about that page’s relation to other pages, and in some cases all or a portion of the media available on the page as well. Search engines need to index all of this information so that they can run searches on it efficiently, rather than having to run around the Internet every time a search query is sent.

The newest trend in search engines is to move away from keyword-based searches and to embrace concept-based searches.
The newest trend in search engines is to move away from keyword-based searches and to embrace concept-based searches.

Search engines create these databases by performing periodic crawls of the Internet. Early versions often required pages to be submitted to them in order to crawl them, but now most pages are found by following links from other pages. What are called robots or spiders, computer programs built to index pages, flit from page to page, recording all of the data on the page, and following every link to new pages. Different search engines refresh their indexes at different intervals, depending on how many spiders they constantly have crawling, and how fast those spiders crawl, with some working their way through the Internet every day or two, and others only doing a periodic refresh every week or month.

As the spider goes through these pages, it records the words it finds on the pages. It makes notes about how many times each word appears, whether the words are weighted in certain ways, perhaps based on size, location, or HTML markup, and decides how relevant the words are based on the links that come in to the page, and on the general context of the page.

Search engines then must weight the value of each page, and the value of each page for the words that appear on it. This is the trickiest part, but also the most important. At the most simple level it could simply keep track of every word on the page, and record that page as relevant for searches with that keyword. This wouldn't do much good for most users, however, as what is desired is the most relevant page for their search query. So different engines come up with different ways of weighting importance.

The algorithms that various search engines use are well protected, to prevent people from specifically creating pages to get better ranks, or at least to limit the degree to which they can do that. This difference is why different engines yield different results for the same terms. Google might determine that one page is the best result for a search term, and Ask might determine that same page is not even in the top 50. This is all just based on how they value inbound and outbound links, the density of the keywords they find important, how they value different placement of words, and any number of smaller factors.

The newest trend in search engines, and likely the future of search in general, is to move away from keyword-based searches to concept-based searches. In this new form of search, rather than limiting a search to the keywords the searcher inputs, the program tries to figure out what those keywords mean, so that it can suggest pages that may not include the exact word, but nonetheless are topical to the search. This is still a developing field, but so far seems to have a lot of potential in making searches more relevant, making the web an even easier place to find exactly what you're looking for.

You might also Like

Discussion Comments


Google Adwords offers some of the best conversion tracking, if not the best conversion tracking, in the world. But guess what? Most businesses advertising on Adwords are not using it. They don’t know how well their ads are performing on a campaign level, nor an ad group level, nor an ad level, nor a keyword level. All they know is clicks – but setting up conversion tracking will enable Adwords to tell you exactly which campaigns and which ad groups and which ads and which keywords are actually becomes leads or sales.

I confess that I spent thousands on Google Adwords without conversion tracking, but now I wouldn’t spend another dollar without it.


Do you know of any webcrawlers, or web search engines, that also crawl and index ALTERNATIVE domain names as .geek, .glue, .music? (domain

names from alternative root DNS, to which hyperlinks exist from 'normal' sites)


Bhutan- Also, remember that if the traffic does not follow the high ranking article then Google or any other search engine would bump it back down and rank a better performing article higher.

There is an art to providing quality content that is also optimized for the search engine marketing.

Sometimes search engine listings are more easily found on Google. I almost never have a problem finding what I need with Google. The only thing that I do prefer with Bing is the maps. The Bing maps are much better than Google maps. They are more detailed and specific than Google.


SurfNturf- I just want to say that these search engine listings rank data by conducting key word searches.

These key word searches identify what the media is about as well as how the search engine ranks it in terms of traffic.

Words that attract more attention that are readily available on the document provide higher rankings.

For example, if an article is submitted via the internet with high traffic keywords that it also well researched with quality sources will probably rank on the first page of Google.

This makes the article more readily available. The use of strategically placed target words is an art referred to as search engine optimization.


Anon105709- I agree with you. The writer really lays out the mystery of search engines well.

For me the top web search engine is Google. Google is the largest search engine which is the one most people use when they provide search engine submissions. A list of search engines aside from Google includes Yahoo, AOL, and Bing.


Wow, that's really cool. I had no idea how search engines worked. Concept-based searches sound like an excellent idea, as well. I already expand to synonyms pretty quickly, if I can't find what I want, but this sounds even broader.

Post your comments
Forgot password?