Working of Search Engines

The first basic truth you need to learn about SEO is that search engines are not humans. While this might be obvious for everybody, the differences between how humans and search engines view web pages aren't. Unlike humans, search engines are text-driven. Although technology advances rapidly, search engines are far from intelligent creatures that can feel the beauty of a cool design or enjoy the sounds and movement in movies. Instead, search engines crawl the Web, looking at particular site items (mainly text) to get an idea what a site is about. This brief explanation is not the most precise because as we will see next, search engines perform several activities in order to deliver search results – crawling, indexing, processing, calculating relevancy, and retrieving.

First, search engines crawl the Web to see what is there. This task is performed by e piece of software, called a crawler or a spider (or Googlebot, as is the case with Google). Spiders follow links from one page to another and index everything they find on their way. Having in mind the number of pages on the Web (over 20 billion), it is impossible for a spider to visit a site daily just to see if a new page has appeared or if an existing page has been modified. Sometimes crawlers will not visit your site for a month or two, so during this time your SEO efforts will not be rewarded. But there is nothing you can do about it, so just keep quiet.

What you can do is to check what a crawler sees from your site. As already mentioned, crawlers are not humans and they do not see images, Flash movies, JavaScript, frames, password-protected pages and directories, so if you have tons of these on your site, you'd better run theSpider Simulator below to see if these goodies are viewable by the spider. If they are not viewable, they will not be spidered, not indexed, not processed, etc. - in a word they will be non-existent for search engines.

After a page is crawled, the next step is to index its content. The indexed page is stored in a giant database, from where it can later be retrieved. Essentially, the process of indexing is identifying the words and expressions that best describe the page and assigning the page to particular keywords. For a human it will not be possible to process such amounts of information but generally search engines deal just fine with this task. Sometimes they might not get the meaning of a page right but if you help them by optimizing it, it will be easier for them to classify your pages correctly and for you – to get higher rankings.

When a search request comes, the search engine processes it – i.e. it compares the search string in the search request with the indexed pages in the database. Since it is likely that more than one pages (practically it is millions of pages) contains the search string, the search engine startscalculating the relevancy of each of the pages in its index to the search string.

There are various algorithms to calculate relevancy. Each of these algorithms has different relative weights for common factors like keyword density, links, or metatags. That is why different search engines give different search results pages for the same search string. What is more, it is a known fact that all major search engines, like Yahoo!, Google, MSN, etc. periodically change their algorithms and if you want to keep at the top, you also need to adapt your pages to the latest changes. This is one reason (the other is your competitors) to devote permanent efforts to SEO, if you'd like to be at the top.

The last step in search engines' activity is retrieving the results. Basically, it is nothing more than simply displaying them in the browser – i.e. the endless pages of search results that are sorted from the most relevant to the least relevant sites.

What Is SEO

Search Engine Optimization (SEO) is often considered the more technical part of Web marketing. This is true because SEO does help in the promotion of sites and at the same time it requires some technical knowledge – at least familiarity with basic HTML. SEO is sometimes also called SEO copyrighting because most of the techniques that are used to promote sites in search engines deal with text. Generally, SEO can be defined as the activity of optimizing Web pages or whole sites in order to make them more search engine-friendly, thus getting higher positions in search results.

One of the basic truths in SEO is that even if you do all the things that are necessary to do, this does not automatically guarantee you top ratings but if you neglect basic rules, this certainly will not go unnoticed. Also, if you set realistic goals – i.e to get into the top 30 results in Google for a particular keyword, rather than be the number one for 10 keywords in 5 search engines, you will feel happier and more satisfied with your results.

Although SEO helps to increase the traffic to one's site, SEO is not advertising. Of course, you can be included in paid search results for given keywords but basically the idea behind the SEO techniques is to get top placement because your site is relevant to a particular search term, not because you pay.

SEO can be a 30-minute job or a permanent activity. Sometimes it is enough to do some generic SEO in order to get high in search engines – for instance, if you are a leader for rare keywords, then you do not have a lot to do in order to get decent placement. But in most cases, if you really want to be at the top, you need to pay special attention to SEO and devote significant amounts of time and effort to it. Even if you plan to do some basic SEO, it is essential that you understand how search engines work and which items are most important in SEO.

Differences Between the Major Search Engines

The basic principle of operation of all search engines is the same, the minor differences between them lead to major changes in results relevancy. For different search engines different factors are important. There were times, when SEO experts joked that the algorithms of Yahoo! are intentionally made just the opposite of those of Google.

 While this might have a grain of truth, it is a matter a fact that the major search engines like different stuff and if you plan to conquer more than one of them, you need to optimize carefully.There are many examples of the differences between search engines. 

For instance, for Yahoo! and MSN, on-page keyword factors are of primary importance, while for Google links are very, very important. Also, for Google sites are like wine – the older, the better, while Yahoo! generally has no expressed preference towards sites and domains with tradition (i.e. older ones). Thus you might need more time till your site gets mature to be admitted to the top in Google, than in Yahoo!.

SEO Optimization and JavaScript

You might find a lot of articles related to using JavaScript Code or not for better SEO Optimization. Some might advise not to use it at all, while some advise to use it within certain restrictions. For example, "don't use long JavaScript codes", "use them at the bottom of your page " etc.
First of all make it very clear - do not fear putting the JavaScript codes if they are really required. In no way it's going to disrupt you in SEO Optimization. The point is WHERE to keep the codes.
This has been a topic of debate since a long time now as far as SEO optimization is concerned. Do the codes really mater? Does the placement of codes really matter?


Lets discuss where we usually keep the JavaScript codes. Most of the webmasters prefer inserting the JavaScript codes at the beginning of the HTML file, between the and section of the file.
It goes like this:


The Body of the Page!
Irrespective of the length of the code it's always preferred there. The question is WHY? Is it really required to be kept there as a software rule, or the codes can be kept anywhere in the file?
The JavaScript codes can be kept ANYWHERE in the file. They will work as fine. So why do the webmasters prefer keeping it there? It's probably to maintain the clarity of the page. Moreover if the page gets too long and the codes are scattered throughout, it becomes difficult to find out the exact JavaScript code that has to be debugged or that has to be changed. This is a kind of convention too that is being followed by most of the webmasters. So the placement of the JavaScript codes is just a customary practice rather than a rule and it has nothing to do with SEO optimization.


The search engines have a mammoth task of spidering millions of ever growing web pages. Google has already reached the 3 billion mark. They are really busy and do not want to be loaded with any more work.


Imagine a web page that has just 2 lines of text or information but 120 lines of JavaScript Code. For getting those 2 lines of information the search engines have to pass through all the unnecessary 120 lines of code. Keeping in mind what is written above do you think the search engines will like the page? They are very busy and if for 2 lines of information they have to work so hard then it's unlikely that this page will ever get a good ranking. How much ever hard you try to optimize it.


Many SEO gurus have come out with an idea to tackle this problem to optimize such a page. According to them putting the JavaScript code at the bottom of the page will solve all the problems since the search engines will not have to go farther down to search the 2 lines of information.


This is what they say:


The Body of the Page!
Note that in this situation the body of the page is above the JavaScript Code. According to the SEO gurus the search engines will now get the content of the page before the code and therefore the importance of the page increases. Right? WRONG!What if there were 2 more lines of content right at the bottom of the page just after the tag? The search engines will have to come to the bottom of the page to get those 2 lines of information. Even if you do not have any information at the bottom of the page the search engines do spider the whole page. They don't have any idea whatsoever whether you have included some content at the bottom of your page or not. The search engines look for tag or end of page whichever comes earlier to end their crawling. So they crawl your whole page anyway irrespective of where you have inserted the JavaScript code and will obviously rank your page on the basis of the content found. This again brings us to square one. Where should you include the JavaScript code if at all?


Those who know this scripting language know very well that a different file can be created to keep all the JavaScript codes separately. This file has an extension of .js. It is a simple file that can be created using notepad. It has nothing except the JavaScript code. NO HTML tags are included. You can place functions, code and style sheets into this external Js file and refer to the file with one line from each web page that wants to use the functions contained in it.
This .js file is kept in the same directory where the file in respect of whom the codes were written resides. Whenever the required codes are called they are taken from the .js file and executed. This file is called only once and then taken from the cache memory of your computer for further execution. It is retrieved only when the cache memory is expired.




This is how an external JavaScript file is included in the page:


The Body of the Page!


This process has a few advantages. External JavaScript files saves bandwidth, which can be achieved by caching any JavaScripts and Cascading Style Sheets (CSS) that are repeated often in a web site. Secondly the length of the page reduces giving way to faster download.
You can always instruct a search engine to index a page or not using a robots.txt file. Once the .js file is ready and uploaded to the server you can always tell the search engines NOT to index your external JavaScript file allowing them to save their time. If you don't know how to write robots.txt file, read article on creating robots.txt file.




This technique solves both the purposes - the search engines do not have to look for the information, they get it without wasting a second and your purpose is solved too - you give search engines the important information fast!


you need not bother if your page is full of Unique and Valuable information, or if you are providing Quality services. The JavaScript code is a trivial matter if your site is content rich. Secondly, the search engines actually do not spider the JavaScript codes. As soon as they find the tag. But they have to go through the codes. So, by including the JavaScript code you are only taking a little bit of their time. Yes, too much of JavaScript code with very less content may irritate them. This is the only reason why a page with a lot of code with comparatively less content does not get a good Google PR (page rank). Unfortunately it's the JavaScript code that has to take all the blame.


Truly speaking there is an unnecessary confusion being created with the use of JavaScript code vis-à-vis SEO optimization. Give some useful content and great service, your JavaScript codes will then play virtually no role in SEO optimization.